Transform-based modal heterogeneous abstract sentiment analysis system

Through the Transformer-based modal heterogeneity abstract sentiment analysis system, text and speech features are extracted using graph convolution network and attention mechanism, the problem of modal heterogeneity and insufficient information utilization in the existing technology is solved, and efficient Accurate sentiment analysis.

CN118916476AActive Publication Date: 2024-11-08NORTHEAST FORESTRY UNIV

Patent Information

Application Number
CN202410958527.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-08
Estimated Expiration
2044-07-17

AI Technical Summary

Technical Problem

The existing multimodal sentiment analysis methods ignore the heterogeneity and ambiguousness between modals, resulting in insufficient accuracy and robustness of sentiment analysis, and it is difficult to effectively extract and utilize long text information and multiple texts in text. Document information.

Method used

Using a modal heterogeneous abstract sentiment analysis system based on Transformer, including a pre-trained feature extraction module, a multimodal abstract generation module and a modal fusion module, text and speech features are extracted through graph convolution network and attention mechanism to construct inter-sentence Similarity graphs are performed, subject segmentation and graph coding are performed, and sentiment analysis is performed in combination with multimodal fusion mechanism.

Benefits of technology

It realizes efficient, accurate and robust sentiment analysis, fully explores complementary information from different modes, and improves the accuracy of emotion recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916476A_ABST
    Figure CN118916476A_ABST
Patent Text Reader

Abstract

The invention discloses a modal heterogeneous abstract sentiment analysis system based on Transform, and belongs to the technical field of information mining. The invention aims to solve the problem of poor sentiment analysis accuracy caused by insufficient mining of complementary information in a sentiment analysis method of a traditional modal fusion mode. The system comprises a pre-training feature extraction module, a multi-modal abstract generation module and a modal fusion module. Wherein the pre-training feature extraction module comprises a text feature extraction unit for marking and coding a text, and a voice feature extraction unit for discretizing and coding voice; the multi-modal abstract generation module firstly encodes paragraphs, constructs an undirected weighted graph representing similarity between sentences, divides sub-graphs based on clustering of a graph convolutional network, and encodes the sentences by using graph structure information; fusing the feature vectors obtained by the graph coding layer and the paragraph coding layer and generating an abstract sequence; the modal fusion module performs sentiment classification based on the abstract text sequence and the speech features.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of information mining and relates to a sentiment analysis system. Background Art

[0002] With the rapid advancement of artificial intelligence technology, people's requirements for intelligent human-computer interaction scenarios are constantly increasing. In real life, multimodal data usually consists of three modalities: vision (image), acoustics (speech), and transcribed text. Since the expression of human emotions is uncertain and diverse, multimodal sentiment analysis methods can combine complementary information between multiple information sources such as text, speech, and images to provide more comprehensive and accurate emotion recognition results, allowing machines to understand human emotions in a way that is more in line with human habits. Therefore, modal heterogeneous summary sentiment analysis methods have broad application prospects in many fields, but the current research on modal heterogeneous summary sentiment analysis methods still has some problems and is difficult to meet the needs in complex environments.

[0003] Traditional emotion recognition methods mostly use low-level features, such as multi-layer multi-head fusion attention mechanism and multi-modal recognition method of speech emotion recognition based on recurrent neural network (RNN), so there is still a lot of room for improvement and development in multi-modal recognition ability. Existing multi-modal sentiment analysis methods usually adopt simple modal fusion methods, such as early fusion, mid-term fusion or late fusion. These methods ignore the heterogeneity and misalignment between different modalities, resulting in the loss of complementary information and the introduction of noise information, thus affecting the accuracy and robustness of sentiment analysis. In addition, the introduction of Transformer has made a breakthrough in generative text summarization, but the semantic relationship between sentences and paragraphs in the original text needs to be improved. Simple feature concatenation and feature fusion may cause a large loss of some modal data, thereby reducing the difference between features and resulting in low recognition accuracy. Multi-modal sentiment analysis methods usually do not take into account the long text information and multi-document information in text data. This information is very important for understanding the main idea and emotion of the text, but due to its complexity and diversity, it is difficult to be effectively extracted and utilized.

[0004] For the above reasons, in an environment where the amount of network data is growing rapidly, it is crucial to provide an automatic, accurate and multi-modal summary sentiment analysis method. Summary of the invention

[0005] The present invention aims to solve the problem that the sentiment analysis method of the traditional modal fusion method has insufficient mining of complementary information, which in turn leads to the problem of poor accuracy of sentiment analysis.

[0006] The Transformer-based modality heterogeneous summary sentiment analysis system includes a pre-trained feature extraction module, a multi-modal summary generation module and a modality fusion module;

[0007] The pre-training feature extraction module includes a text feature extraction unit and a speech feature extraction unit;

[0008] Text feature extraction unit: mark the text and encode the marked text to obtain X α ; By feature information X a The represented document data is sent to a multimodal summary generation module;

[0009] Speech feature extraction unit: discretize the speech and encode the discretized speech to obtain X β ;

[0010] The processing process of the multimodal summary generation module includes the following steps:

[0011] S201, dividing the document data set into paragraphs and inputting them into a paragraph encoding layer, the paragraph encoding layer uses Transformer to encode the paragraphs, extracting sentences from the output of the paragraph encoding layer, and constructing an undirected weighted graph G representing the similarity between sentences, which represents the similarity relationship between sentences in the summary text; performing a clustering algorithm based on a graph convolutional network on the graph G, and dividing the graph G into several subgraphs, each of which represents a topic or a viewpoint;

[0012] S202, inputting the subgraph of the undirected weighted graph into a graph encoding layer, and encoding the sentence using the graph structure information;

[0013] S203, the feature vectors obtained from the graph encoding layer and the paragraph encoding layer are fused and input into a sequence-to-sequence model with an attention mechanism and a pointer network to generate a summary sequence; the summary sequence generated by the multimodal summary generation module is recorded as

[0014] The processing process of the modality fusion module includes the following steps:

[0015] Summary text sequence for feature matrix representation and voice X β ,get and The Query, Keys and Values ​​matrices corresponding to text and speech respectively. are the corresponding weight matrices respectively;

[0016] Through multimodal fusion mechanism Get from β to α ′The cross-modal Attention output of is expressed as Right now:

[0017]

[0018] in, is the scale coefficient, and have the same length;

[0019] In the same way, through the multimodal fusion mechanism Get from α ′ The cross-modal Attention output to β is expressed as

[0020] Use X con express Y β One of, or Y β The fusion features of X con After a one-dimensional convolutional layer, we get Then it is input into the attention-based LSTM module; then it is sent to the fully connected layer and outputs the sentiment result after softmax processing.

[0021] Furthermore, the text feature extraction unit uses the GPT-2 tagger to tag the text, and uses the RoBERTa model to encode the tagged text to obtain X α .

[0022] Furthermore, the speech feature extraction unit discretizes the speech using the VQ-Wav2Vec model, and encodes the discretized speech using the Speech-BERT model to obtain X β .

[0023] Furthermore, the paragraph encoding layer described in S201 is composed of several Local Transformer Layers and Global Transformer Layers, and the Local Transformer Layers and Global Transformer Layers are stacked alternately to form a deep Transformer network, and each Transformer Layer includes a Multi-Head Attention sublayer and a feedforward neural network sublayer, wherein Multi-Head Attention is a multi-head attention mechanism.

[0024] Furthermore, the specific process of S201 includes the following steps:

[0025] (1) Assume that the document data set contains n1 documents d1-d n1 , each document consists of m paragraphs; each document is segmented to obtain the set of all paragraphs P = {P1, P2, P3, ... P m}, where P i Indicates the i-th paragraph;

[0026] (2) For each paragraph P i , perform word segmentation on it and get the set W of all words i ={w1,w2,…,w n1}, where w k represents the jth word;

[0027] (3) For each word w j Use the word embedding model to get its word vector e j , and then use the position encoding model to get its position vector Add the two together to get the word representation vector x j ,Right now

[0028] (4) Change paragraph P i The representation vectors x of all words in ij Form a matrix X∈R n*d , where n is the number of words in the paragraph and d is the dimension of the representation vector;

[0029] (5) The matrix X is used as the input of the paragraph encoding layer and first input into the Local Transformer Layer, where it is locally self-attention encoded to obtain the encoding matrix Y of the paragraph.

[0030] (6) Input the matrix Y into the Global Transformer Layer to obtain the context vector z for each paragraph i ;

[0031] (7) For each paragraph, the context vector z i Input a two-layer feedforward network to obtain the summary vector s for each paragraph i ;

[0032] (8) Summary vector s for each paragraph i Input a softmax layer to get the summary probability prob of each paragraph i ∈[0,1], where prob i Indicates paragraph prob i The probability of being selected into the abstract; specifically expressed as:

[0033]

[0034] Where w∈R d represents the weight vector of the softmax layer, b∈R represents the bias scalar of the softmax layer, and m represents the number of paragraphs in the document;

[0035] (9) Based on the summary probability prob of each paragraph i , select several paragraphs as candidate paragraphs of the summary, and splice the candidate paragraphs in the order of the original document to form a summary text of the paragraph;

[0036] (10) An undirected weighted graph is constructed from the output vector of the paragraph encoding layer to calculate the cosine similarity matrix between sentences. The specific steps are as follows:

[0037] The encoding matrix of each paragraph in the summary text is recorded as Y i , the encoding matrix Y for each paragraph i , use the sentence segmentation model to process it into sentences, and get the sentence set S = {s1, s2, ..., s k}, where s j represents the jth sentence, k is the number of sentences in the paragraph;

[0038] For each sentence s j , use the sentence encoding model to get its sentence vector v j , and then use the position encoding model to get its position vector Add the two together to get the sentence representation vector xos j ,Right now

[0039] The representation vectors xos of all sentences in the paragraphs j Form a matrix XOS;

[0040] For every two sentences xos j ,xos k , calculate the cosine similarity sim(xos j ,xos k ), and judge whether two sentences are connected according to a threshold value θ; then, with sentences as nodes and similarities between sentences as edge weights, an undirected weighted graph G = (V, E) is constructed to represent the similarity relationship between sentences in the summary text;

[0041] The graph G is processed by a clustering algorithm based on a graph convolutional network to dynamically calculate the attention weights between nodes and divide the graph G into several subgraphs G1, G2, …, G l , where G i =(V iE i ), V i It's G i The set of midpoints, E i It's G i The set of edges in each subgraph G i Represents a theme or a point of view.

[0042] Furthermore, the processing of the Local Transformer Layer includes:

[0043] Based on the input matrix composed of word embedding vectors, the input tokens are embedded and positionally encoded to obtain the initial representation of each token; the multi-head self-attention mechanism Multi-HeadAttention is applied to the representation of each token to obtain the attention score of each token to other tokens in the paragraph; the attention scores are residually connected; then layer normalization is performed, and a feedforward neural network is applied to obtain the final representation of each token, and the final representation is residually connected; finally, the output representation of each token is obtained through layer normalization.

[0044] Furthermore, the matrix Y is input into the Global Transformer Layer to obtain the context vector z for each paragraph i , the specific process is as follows:

[0045] z i =LayerNorm(g i +y ij )

[0046] where z i ∈R d , z i is the i-th row of Y; LayerNorm represents the layer normalization operation; g i Indicates header information c i With each input token vector y ij The vector obtained after addition is:

[0047]

[0048] in, and denote the weight matrices of the first and second layers of the feedforward neural network sublayers, c i It represents the sum of the context vectors of the ith paragraph after all the heads of the Inter-paragraph Attention mechanism are passed through:

[0049]

[0050] Among them, h represents the number of heads, It represents the context vector obtained after the i-th paragraph passes through the Inter-paragraph Attention mechanism of the z-th head. The T in represents the transpose operation. represents the learnable parameter matrix in the Inter-paragraph Attention mechanism of the z-th head, Represents vector query, key and value vectors, which are obtained from y ij Linear transformation.

[0051] Furthermore, the graph encoding layer in step S202 is composed of multiple Graph Encoding Layers, and the Graph Encoding Layer is implemented using technologies such as a graph attention mechanism or a graph neural network. The specific steps are as follows:

[0052] Subgraph G of the undirected weighted graph i Input Graph Encoding Layer, perform graph encoding on each subgraph to obtain node features and edge features. Node features are represented by matrix V, and edge features are represented by matrix E. Each row of E is E i The feature vector representing the i-th edge is obtained by fusing node features and edge features into a graph representation vector, using an enhanced graph based on a graph attention network mechanism to calculate the correlation between nodes in the vector and update the feature vector of the node. That is, the explicit graph representation is input into the encoding process through the graph attention network mechanism, and the explicit relationship in the graph is used to learn better inter-sentence relationships. Each sentence collects information from other related sentences, thereby obtaining global information from the entire input.

[0053] set up represents the output vector of the i-th sentence at the t-1th graph encoding layer, where is the input sentence vector; for each node i, the context representation u i is the vector representation of the ith node, obtained by the following steps:

[0054] α ij =softmax(e ij +R ij )

[0055]

[0056] Among them, α ij represents the weight coefficient between the i-th node and the j-th node; W K ,W Q ,WV is the parameter weight, which is used to calculate the parameter matrix of query, key and value respectively; e ij represents the potential relationship weight between the i-th node and the j-th node, d head represents the dimension of each attention head, R ij Represents the relative position information between the i-th node and the j-th node;

[0057] Finally, u′ i After two layers of feedforward neural network, the feature vector u is output i .

[0058] Furthermore, R ij Determined by calculating the connection between two nodes in the undirected weighted graph G:

[0059]

[0060] Among them, σ represents the standard deviation of the strength of the graph structure influence; σ is a hyperparameter; G[i][j] is the element in the undirected weighted graph G.

[0061] Furthermore, the specific process of S203 includes the following steps:

[0062] S2031, the feature matrix u i and i As the input of the encoder, a bidirectional LSTM or GRU is used to encode the feature matrix to obtain the hidden state matrix H, where each row represents the hidden state of a node or token;

[0063] S2032. At the decoder end, a unidirectional LSTM or GRU is used to generate a summary sequence. The initial state is the last hidden state of the encoder, and the initial input is a special start symbol. <start>;

[0064] S2033, according to the output probability distribution p iut , select the word or token with the highest probability as the output of the current decoding step;

[0065] S2034, if the output word or token is a special end symbol <end>, then stop decoding, otherwise use the output word or token as the input of the next decoding step, and repeat the above operation until the maximum decoding length is reached or the end symbol is output;

[0066] The summary sequence generated by the multimodal summary generation module is recorded as

[0067] Beneficial effects:

[0068] The present invention proposes a new multimodal summary sentiment analysis system based on Transformer and attention mechanism, which can extract features from multiple modalities such as text and speech on the basis of effectively retaining the low-level features of the data, and generate concise, accurate and coherent summaries using graph structure information and Transformer model, and then use the bidirectional Transformer fusion network to perform modal fusion, which can fully mine the complementary information of different modalities and realize efficient, accurate and robust sentiment analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is a flowchart of the Transformer-based multimodal summary sentiment analysis process. DETAILED DESCRIPTION

[0070] The present invention is described below in conjunction with specific embodiments.

[0071] Specific implementation method 1: Combination Figure 1 To explain this embodiment,

[0072] This embodiment is a Transformer-based modality heterogeneous summary sentiment analysis system, including a pre-training feature extraction module, a multimodal summary generation module and a modality fusion module; wherein,

[0073] Pre-trained feature extraction module: extracts feature vectors from multiple modalities such as text and speech, and uses them as input for the next module. This module uses a self-supervised learning method to learn general image features from large-scale unlabeled data, and uses a pre-trained language model to learn semantic features from text data, which reduces the cost of collecting and annotating large-scale data sets and improves the generalization and migration capabilities of feature representation.

[0074] Multimodal summary generation module: This module is responsible for generating summary sequences from the text feature vectors obtained from the pre-trained feature extraction module and using them as input for the next module. This module adopts a generative summary method based on the Transformer model, uses a document-level sentence relationship graph to capture long text information in multiple documents, and uses a graph attention network mechanism to perform attention calculations on paragraphs and documents, optimizes the summary generation process, and enables the model to generate concise, coherent, and accurate summary text.

[0075] Modal fusion module: This module is responsible for sentiment analysis from the speech feature vector obtained by the pre-trained feature extraction module and the summary sequence obtained by the multimodal summary generation module, and outputs the sentiment category and confidence. This module uses a bidirectional Transformer fusion network and uses collaborative learning to build bidirectional interactions to achieve robustness to missing modalities and improve the accuracy and robustness of sentiment analysis.

[0076] More specifically,

[0077] (I) Pre-trained feature extraction module:

[0078] Human language is a multimodal information carrier, which includes not only natural language text, but also non-language modal information such as facial expressions and sounds. Therefore, in the data preprocessing stage, the pre-trained feature extraction module uses different methods to extract feature vectors from text and speech modalities; the processing process of the pre-trained feature extraction module includes a text feature extraction unit and a speech feature extraction unit:

[0079] For text data, use the GPT-2 tagger to tag the text, and use the RoBERTa model to encode the tagged text to get X α , through continuous alternating training, GPT-2 can gradually improve the coherence, relevance, and diversity of labeled text, and RoBERTa can gradually improve its ability to distinguish between real text and generated text;

[0080] For speech data, the VQ-Wav2Vec model is used to discretize the speech, and the Speech-BERT model is used to encode the discretized speech to obtain X β .

[0081] In actual use, the speech signal and text are input into their respective pre-trained language models. When both speech and text modalities are labeled, they are sent to subsequent processing. α The document data represented by (processed by the pre-trained model RoBERTa) is sent to the multimodal summary generation module.

[0082] (II) Multimodal summary generation module:

[0083] S201, divide the document data set into paragraphs and input a paragraph encoding layer, the paragraph encoding layer uses Transformer to encode the paragraphs, extract sentences from the output of the paragraph encoding layer, and construct an undirected weighted graph G representing the similarity between sentences, which represents the similarity relationship between sentences in the summary text; perform a clustering algorithm based on a graph convolutional network on the graph G, and divide the graph G into several subgraphs G1, G2, ..., G l , each subgraph G i Represents a theme or a point of view.

[0084] The paragraph encoding layer is composed of several Local Transformer Layers and Global Transformer Layers, which are stacked alternately to form a deep Transformer network. Each Transformer Layer contains a Multi-Head Attention sublayer and a feedforward neural network sublayer. The Multi-Head Attention sublayer allows the model to learn the dependency between any two positions in the input sequence, and the feedforward neural network sublayer allows the model to learn the nonlinear transformation of each position in the input sequence. The input and output of both the Multi-Head Attention sublayer and the feedforward neural network sublayer are matrices X∈R composed of word embedding vectors. n*d , each word embedding vector x ij The dimension is d model .

[0085] S202, inputting the undirected weighted graph into a graph coding layer, and encoding the sentence using the graph structure information;

[0086] S203: Fuse the feature vectors obtained by the graph encoding layer and the paragraph encoding layer, and input them into a sequence-to-sequence model with an attention mechanism and a pointer network to generate a summary sequence.

[0087] in,

[0088] The specific process of S201 is as follows:

[0089] (1) Assume that the document data set contains n1 documents d1-d n1 , each document consists of m paragraphs; each document is segmented to obtain the set of all paragraphs P = {P1, P2, P3, ... P m }, where P i Indicates the i-th paragraph;

[0090] (2) For each paragraph P i , perform word segmentation on it and get the set W of all words i ={w1,w2,…,w n1 }, where w j represents the jth word;

[0091] (3) For each word w j Use the word embedding model to get its word vector e j , and then use the position encoding model to get its position vector Add the two together to get the word representation vector x j ,Right now

[0092] (4) Change paragraph P i The representation vectors x of all words in ij Form a matrix X∈R n*d , where n is the number of words in the paragraph and d is the dimension of the representation vector;

[0093] (5) The matrix X is used as the input of the paragraph encoding layer and first input into the Local Transformer Layer, where it is locally self-attention encoded to obtain the encoding matrix Y of the paragraph.

[0094] The Local Transformer Layer allows each token to pay attention to the contextual information within the same paragraph. The Local Transformer Layer includes the Multi-Head Attention sublayer and the feed-forward neural network FFN sublayer. Multi-Head Attention is the introduced multi-head attention mechanism, namely MultiHead (Q, K, V), which allows each token to pay attention to other tokens with different attention distributions; in short, the Local Transformer Layer is mainly used to encode the contextual information of each token within the paragraph:

[0095] The input token is embedded and positionally encoded to obtain the initial representation of each token.

[0096] Apply a multi-head self-attention mechanism to the representation of each token to obtain the attention score of each token to other tokens in the paragraph; perform a residual connection on the attention scores;

[0097] Then, layer normalization is performed, and a feed-forward network is applied to obtain the final representation of each token, and residual connections are performed on the final representation.

[0098] Finally, the output representation of each token is obtained through layer normalization.

[0099] For the mth Local Transformer Layer, the input is X (m-1) , the output is X m , where X (m-1) and X m They are all matrices composed of word embedding vectors. Each row represents the vector representation of a token. The dimension of the embedding representation of a token is d. model , X (m-1) Represents the embedding representation of a paragraph, its dimension is n*d model , where n is the number of tokens in the paragraph.

[0100] The specific processing process of each Local Transformer Layer is as follows:

[0101] 5.1. X (m-1) Linear transformation yields Q, K, and V, where Q, K, and V represent query, key, and value vectors, respectively.

[0102] Apply linear transformations respectively to obtain z groups Q, K, V, each with a dimension of d model .

[0103] 5.2. Apply scaled dot product attention to each set of Q, K and V, and get z output vectors, each of which has a dimension of d model The calculation formula for scaled dot product attention is as follows:

[0104]

[0105] Among them, Q, K, V are from X (m-1) It is derived from linear transformation, T represents the transposition operation, d represents the dimension of the word embedding vector, and softmax is a normalization function used to calculate the attention weight of each token to other tokens.

[0106] 5.3. The vector obtained after the i-th paragraph passes through the Multi-Head Attention sublayer of the z-th head is the output vector of each head The calculation is as follows

[0107]

[0108] Among them, W i Q ,W i K ,W i V ,W O is the learnable parameter matrix.

[0109] Concatenate the output vectors of multiple heads

[0110] MultiHead(Q,K,V)=Concat(head1,…,head h )W O

[0111] Where h is the number of heads.

[0112] 5.4. For each token in the mth Local Transformer Layer, the calculation is as follows:

[0113]

[0114] The calculation of the Feed-Forward Network is as follows:

[0115] FFN(h ij )=max(0,h ij W1+b1)W2+b2

[0116] The output vector of the Local Transformer Layer is:

[0117] y ij =LayerNorm(h ij +FFN(h ij ))

[0118] in, b2∈R d is the learnable parameter matrix and bias vector, h ij ∈R d Represents the output vector after the MultiHead sublayer in the m-th Local Transformer Layer; max is the element-wise maximum function, used to increase the nonlinearity of the model. ff Refers to the output dimension of the first layer of the feedforward neural network sublayer. It is an adjustable hyperparameter used to control the complexity and expressiveness of the model.

[0119] Among them, LayerNorm is the layer normalization function, defined as follows:

[0120]

[0121] Among them, γ∈R d , β∈R d is a learnable parameter vector, μ∈R d ,σ∈R d is the mean and standard deviation vector of X in the last dimension, which is used to eliminate the offset and scaling between different layers.

[0122] The encoding vector y of all paragraphs ij The matrix Y∈R m*d .

[0123] (6) Input the matrix Y into the Global Transformer Layer to obtain the context vector z for each paragraph i , the specific process is as follows:

[0124] z i =LayerNorm(g i +y ij )

[0125] where z i ∈R d , z i is the i-th row of Y; LayerNorm represents the layer normalization operation, which is used to alleviate the problem of gradient disappearance or explosion and improve the stability and convergence speed of the model. i Indicates header information c i With each input token vector y ij The vector obtained after addition is:

[0126]

[0127] in, and denote the weight matrices of the first and second layers of the feedforward neural network sublayers, c i It represents the sum of the context vectors of the i-th paragraph after the Inter-paragraph Attention mechanism of all heads, that is:

[0128]

[0129] Among them, h represents the number of heads, It represents the context vector obtained after the i-th paragraph passes through the Inter-paragraph Attention mechanism of the z-th head. It is the weighted sum of the value vectors of all paragraphs. The weight is calculated by the dot product attention function between the query and the key vector. It is specifically expressed as:

[0130]

[0131] Where T represents the transpose operation, represents the learnable parameter matrix in the Inter-paragraphAttention mechanism of the z-th head, which is used to map the input vector into query, key, and value vectors, Represents Query, Key, Value, that is, vector query, key and value vector, which is from y ij The specific calculation is as follows:

[0132]

[0133] y ij It represents the vector obtained after the i-th paragraph is processed by the Local Transformer Layer, which is X ij The output vector of the last word of Represents the representation matrix of the jth word in the i-th paragraph, where n is the number of words in the paragraph and d is the dimension of the word embedding vector.

[0134] (7) For each paragraph, the context vector z i Input a two-layer feedforward network to obtain the summary vector s for each paragraph i ∈R t*d , where s i Yes i A nonlinear transformation of , t is the number of tokens in the paragraph;

[0135] s i =W2ReLU(W1z i +b1)+b2

[0136] in, Represent the weight matrices of the first and second layers of the feedforward neural network sublayers, Denote the bias vectors of the first and second layers respectively, d ff represents the hidden layer dimension of the feedforward neural network, and ReLU represents the linear rectifier unit activation function.

[0137] (8) Summary vector s for each paragraph i Input a softmax layer to get the summary probability prob of each paragraph i ∈[0,1], where prob i Indicates paragraph prob i The probability of being selected into the abstract; specifically expressed as:

[0138]

[0139] Where w∈R d represents the weight vector of the softmax layer, b∈R represents the bias scalar of the softmax layer, and m represents the number of paragraphs in the document.

[0140] (9) Based on the summary probability prob of each paragraph i , select several paragraphs as candidate paragraphs of the summary, and splice the candidate paragraphs together in the order of the original document to form a summary text of the paragraph.

[0141] (10) An undirected weighted graph is constructed from the output vector of the paragraph encoding layer to calculate the cosine similarity matrix between sentences. The specific steps are as follows:

[0142] The encoding matrix Y for each paragraph i , use the sentence segmentation model to process it into sentences, and get the sentence set S = {s1, s2, ..., s k }, where s j represents the jth sentence, k is the number of sentences in the paragraph;

[0143] For each sentence s j , use the sentence encoding model to get its sentence vector v j , and then use the position encoding model to get its position vector Add the two together to get the sentence representation vector xos j ,Right now

[0144] The representation vectors xos of all sentences in the paragraphs j Form a matrix XOS;

[0145] For every two sentences xos j ,xos k , calculate the cosine similarity sim(xos j ,xos k ), and use a threshold θ = 0.5 to determine whether the two sentences are connected, that is, E = {(s j ,s k )|sim(s j ,s k )>0.5}, then take sentences as nodes and the similarity between sentences as edge weights to construct an undirected weighted graph G=(V, E) to represent the similarity relationship between sentences in the summary text;

[0146] The graph G is processed by a clustering algorithm based on a graph convolutional network to dynamically calculate the attention weights between nodes and divide the graph G into several subgraphs G1, G2, …, G l , where G i =(V i E i ), V i It's G i The set of midpoints, E i It's G i The set of edges in each subgraph G i Represents a theme or a point of view.

[0147] The specific process of S202 is as follows:

[0148] Input the undirected weighted graph into the graph encoding layer, use the graph structure information to encode the sentence, and use the attention-based mechanism to encode; the graph encoding layer consists of multiple graph encoding layers. The graph encoding layer is a method that uses graph structure information to encode the input subgraph, which is implemented using graph attention mechanism or graph neural network and other technologies. The specific steps are as follows:

[0149] Subgraph G of the undirected weighted graph i Input the Graph Encoding Layer, perform graph encoding on each subgraph to obtain node features and edge features. Node features are represented by a matrix V of u×d1, where n is the number of nodes in the graph and d1 is the dimension of the node features; edge features are represented by a matrix E of m×d2, where m is the number of edges in the graph and d2 is the dimension of the edge features. Each row of E i Represents the feature vector of the i-th edge. The node features and edge features are fused into a graph representation vector, and the correlation between nodes in the vector is calculated using an enhanced graph based on the Graph Attention Network mechanism, and the feature vector of the node is updated. That is, the explicit graph representation is input into the encoding process through the Graph Attention Network mechanism, and the explicit relationship in the graph is used to learn better relationships between sentences. Each sentence can collect information from other related sentences, thereby obtaining global information from the entire input. Assume represents the output vector of the i-th sentence at the t-1th graph encoding layer, where is the input sentence vector. For each node i, the context representation u i It refers to the vector representation of the i-th node (sentence), which integrates the relationship information of other nodes (sentences) in the node (sentence) and is calculated as the weighted sum of the linearly transformed node (sentence) vectors. The specific calculation is:

[0150] Assume that the length of the input sequence is n s , the dimension of the feature vector is The number of heads is h, then the dimension of each head is

[0151] α ij =softmax(e ij +R ij )

[0152]

[0153] The weighted sum refers to multiplying the value vector of each node (sentence) by a weight coefficient and then summing them up to get the context representation. ij Represents the weight coefficient between the i-th node (sentence) and the j-th node (sentence). W K ,W Q ,W V ∈R d*d are parameter weights, which are used to calculate the parameter matrices of query, key, and value. ij represents the potential relationship weight between the i-th node (sentence) and the j-th node (sentence), d head represents the dimension of each attention head, R ij Represents the relative position information between the i-th node (sentence) and the j-th node (sentence). ij It is obtained by calculating the connection between two nodes (sentences) in the undirected weighted graph G. The main difference of the graph self-attention mechanism is the additional pairwise relationship bias R ij , is calculated by the Gaussian deviation of the weights of the undirected weighted graph G:

[0154]

[0155] Among them, σ represents the standard deviation of the strength of the graph structure influence. σ is a hyperparameter that can be adjusted according to experimental data to achieve the best effect. The larger the value of σ, the weaker the influence of the graph structure, and vice versa. The value of σ can also be determined by grid search or other optimization methods.

[0156] The undirected weighted graph G is a two-dimensional matrix that indicates whether there is a connection between different nodes (sentences) in the graph. If there is a connection between two nodes (sentences), the corresponding position is 1, otherwise it is 0. G[i][h] represents the connection between the i-th node and the j-th node. Gaussian deviation R ij ∈(-inf, 0) measures S i and S j The closeness between them, where the Gaussian deviation R ij is a negative number that measures the closeness between the i-th node and the j-th node. The closer -inf is to 0, the closer the two nodes are to each other; the closer -inf is to negative infinity, the looser the two nodes are. Due to the exponential operation in the softmax function, the Gaussian deviation is approximated by multiplying the latent attention distribution by the weight ∈(0, 1].

[0157] u′ i After two layers of feedforward neural network, the feature vector u is output i ;

[0158] The specific process of S203 is as follows:

[0159] The feature vector u obtained by the graph encoding layer and the paragraph encoding layer i and i The fusion is used as the input of the sequence-to-sequence model. The present invention uses a sequence-to-sequence model with an attention mechanism and a pointer network to generate a summary sequence. Specifically, the sentence vector obtained by the graph encoding layer and the token vector obtained by the paragraph encoding layer are spliced ​​according to the position of the node in the graph to obtain a feature matrix that integrates the graph structure information and the paragraph information. The details are as follows:

[0160] S2031, the feature matrix u i and i As the input of the encoder, a bidirectional LSTM or GRU is used to encode the feature matrix to obtain a (n n +t)×2d hidden state matrix H, where each row represents the hidden state of a node (sentence) or token, where t is the number of tokens in the paragraph, and n is the number of hidden states. n is the number of nodes (sentences) in the graph, d n It is the dimension of node (sentence) features.

[0161] S2032. At the decoder end, a unidirectional LSTM or GRU is used to generate a summary sequence. The initial state is the last hidden state of the encoder, and the initial input is a special start symbol. <start>.

[0162] S2033, according to the output probability distribution p out , select the word or token with the highest probability as the output of the current decoding step.

[0163] S2034, if the output word or token is a special end symbol <end>, then stop decoding, otherwise use the output word or token as the input of the next decoding step, and repeat the above operation until the maximum decoding length is reached or the end symbol is output.

[0164] The summary sequence generated by the multimodal summary generation module is recorded as

[0165] (III) Modal Fusion Module

[0166] In order to achieve bidirectional cross-modal feature information association, the present invention adopts a multimodal fusion mechanism, which can perform two modal α ′ Specifically, assuming that α ′ The modality represents the summary text processed by the multimodal summary generation module, and the β-modality represents the speech. Represent the input feature matrices of summary text and speech respectively, where and T β Respectively represent the number of tokens in the text and the time step of the speech, and d β Represent the feature dimensions of text and speech respectively. In order to perform cross-modal feature interaction and fusion, the attention mechanism (AM) is used, which can calculate the output Attention based on the input Query, Keys and Values. Specifically, let and Represent the Query, Keys, and Values ​​matrices of text and speech respectively, where and Represents the weight matrices of Query, Key and Value for text and speech respectively, d k and d v Represents the feature dimensions of Keys and Values ​​respectively. Through the multimodal fusion mechanism and From β to α ′ Cross-modal Attention output:

[0167] At this time, the output changes from β to α ′ Cross-modal Attention:

[0168]

[0169] in, is the scale coefficient, and have the same length (i.e. );

[0170] From this we can also get that from α ′ Cross-modal Attention to β:

[0171]

[0172] Use X con express Y β One of, or Y β The fusion features are obtained through a one-dimensional convolution layer. Then input into the attention-based LSTM module.

[0173] The calculation process of the one-dimensional convolution layer is as follows:

[0174]

[0175] in, is the size of the modal convolution kernel, d c It is a common dimension of cross-modality.

[0176] Mapping different modal features on the same dimension is conducive to further exploiting the explicit interactions between modalities.

[0177] Finally, it is sent to the fully connected layer and outputs the sentiment result after softmax processing to achieve summary sentiment analysis.

[0178] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.< / end> < / start> < / end> < / start>

Claims

1. A Transformer-based modality-heterogeneous summary sentiment analysis system, characterized by: It includes a pre-trained feature extraction module, a multimodal summary generation module and a modality fusion module; among them, The pre-training feature extraction module includes a text feature extraction unit and a speech feature extraction unit; Text feature extraction unit: mark the text and encode the marked text to obtain X α ; By feature information X α The represented document data is sent to a multimodal summary generation module; Speech feature extraction unit: discretize the speech and encode the discretized speech to obtain X β ; The processing process of the multimodal summary generation module includes the following steps: S201, dividing the document data set into paragraphs and inputting them into a paragraph encoding layer, the paragraph encoding layer uses Transformer to encode the paragraphs, extracting sentences from the output of the paragraph encoding layer, and constructing an undirected weighted graph G representing the similarity between sentences, which represents the similarity relationship between sentences in the summary text; performing a clustering algorithm based on a graph convolutional network on the graph G, and dividing the graph G into several subgraphs, each of which represents a topic or a viewpoint; S202, inputting the subgraph of the undirected weighted graph into a graph encoding layer, and encoding the sentence using the graph structure information; S203, the feature vectors obtained from the graph encoding layer and the paragraph encoding layer are fused and input into a sequence-to-sequence model with an attention mechanism and a pointer network to generate a summary sequence; the summary sequence generated by the multimodal summary generation module is recorded as X α′ ; The processing process of the modality fusion module includes the following steps: For the summary text sequence X represented by the feature matrix α, and voice X β ,get and The Query, Keys and Values ​​matrices corresponding to text and speech respectively. are the corresponding weight matrices respectively; Through the multimodal fusion mechanism CM β→α′ , from β to α ′ The cross-modal Attention output of α′ :=Tran β→α, (X α, ,X β ),Right now: in, is the scale factor, Y α′ With Q α, have the same length; In the same way, through the multimodal fusion mechanism CM α,→β , obtained from α , The cross-modal Attention output to β is denoted as Y β :=Tran α′→β (X β ,X α′ ); Use X con Represents Y α′ , Y β One of, or Y α, , Y β The fusion features of X con After a one-dimensional convolutional layer, we get Then it is input into the attention-based LSTM module; then it is sent to the fully connected layer and outputs the sentiment result after softmax processing.

2. The Transformer-based modality heterogeneous summary sentiment analysis system according to claim 1, characterized in that: The text feature extraction unit uses the GPT-2 tagger to tag the text, and uses the RoBERTa model to encode the tagged text to obtain X α .

3. The Transformer-based modality heterogeneous summary sentiment analysis system according to claim 1, characterized in that: The speech feature extraction unit discretizes the speech using the VQ-Wav2Vec model, and encodes the discretized speech using the Speech-BERT model to obtain X β .

4. The Transformer-based modality heterogeneous summary sentiment analysis system according to claim 1, characterized in that: The paragraph encoding layer described in S201 is composed of several Local Transformer Layers and Global Transformer Layers. The Local Transformer Layers and the Global Transformer Layers are stacked alternately to form a deep Transformer network. Each Transformer Layer contains a Multi-Head Attention sublayer and a feedforward neural network sublayer, where Multi-Head Attention is a multi-head attention mechanism.

5. The Transformer-based modality heterogeneous summary sentiment analysis system according to claim 1, characterized in that: The specific process of S201 includes the following steps: (1) Assume that the document data set contains n1 documents d1-d n1 , each document consists of m paragraphs; each document is segmented to obtain the set of all paragraphs P = {P1, P2, P3, ... P m }, where P i Indicates the i-th paragraph; (2) For each paragraph P i , perform word segmentation on it and get the set W of all words i ={w1,w2,…,w n1 }, where w j represents the jth word; (3) For each word w j Use the word embedding model to get its word vector e j , and then use the position encoding model to get its position vector Add the two together to get the word representation vector x j ,Right now (4) Change paragraph P i The representation vectors x of all words in ij Form a matrix X∈R n*d , where n is the number of words in the paragraph and d is the dimension of the representation vector; (5) The matrix X is used as the input of the paragraph encoding layer and first input into the Local Transformer Layer, where it is locally self-attention encoded to obtain the encoding matrix Y of the paragraph. (6) Input the matrix Y into the Global Transformer Layer to obtain the context vector z for each paragraph i ; (7) For each paragraph, the context vector z i Input a two-layer feedforward network to obtain the summary vector s for each paragraph i ; (8) Summary vector s for each paragraph i Input a softmax layer to get the summary probability prob of each paragraph i ∈[0,1], where prob i Indicates paragraph prob i The probability of being selected into the abstract; specifically expressed as: Where w∈R d represents the weight vector of the softmax layer, b∈R represents the bias scalar of the softmax layer, and m represents the number of paragraphs in the document; (9) Based on the summary probability prob of each paragraph i , select several paragraphs as candidate paragraphs of the summary, and splice the candidate paragraphs in the order of the original document to form a summary text of the paragraph; (10) An undirected weighted graph is constructed from the output vector of the paragraph encoding layer to calculate the cosine similarity matrix between sentences. The specific steps are as follows: The encoding matrix of each paragraph in the summary text is recorded as Y i , the encoding matrix Y for each paragraph i , use the sentence segmentation model to process it into sentences, and get the sentence set S = {s1, s2, ..., s k }, where s j represents the jth sentence, k is the number of sentences in the paragraph; For each sentence s j , use the sentence encoding model to get its sentence vector v j , and then use the position encoding model to get its position vector Add the two together to get the sentence representation vector xos j ,Right now The representation vectors xos of all sentences in the paragraphs j Form a matrix XOS; For every two sentences xos j ,xos k , calculate the cosine similarity sim(xos j ,xos k ), and judge whether two sentences are connected according to a threshold value θ; then, with sentences as nodes and similarities between sentences as edge weights, an undirected weighted graph G = (V, E) is constructed to represent the similarity relationship between sentences in the summary text; The graph G is processed by a clustering algorithm based on a graph convolutional network to dynamically calculate the attention weights between nodes and divide the graph G into several subgraphs G1, G2, …, G l , where G i =(V i E i ), i It's G i The set of midpoints, E i It's G i The set of edges in each subgraph G i Represents a theme or a point of view.

6. The Transformer-based modality heterogeneous summary sentiment analysis system according to claim 4 or 5, characterized in that: The processing of the Local Transformer Layer includes: Based on the input matrix composed of word embedding vectors, the input tokens are embedded and positionally encoded to obtain the initial representation of each token; the multi-head self-attention mechanism Multi-HeadAttention is applied to the representation of each token to obtain the attention score of each token to other tokens in the paragraph; the attention scores are residually connected; then layer normalization is performed, and a feedforward neural network is applied to obtain the final representation of each token, and the final representation is residually connected; finally, the output representation of each token is obtained through layer normalization.

7. The Transformer-based modality heterogeneous summary sentiment analysis system according to claim 4 or 5, characterized in that: Input the matrix Y into the Global Transformer Layer to obtain the context vector z for each paragraph i , the specific process is as follows: z i =LayerNorm(g i +y ij ) where z i ∈R d , z i is the i-th row of Y; LayerNorm represents the layer normalization operation; g i Indicates header information c i With each input token vector y ij The vector obtained after addition is: in, and denote the weight matrices of the first and second layers of the feedforward neural network sublayers, c i It represents the sum of the context vectors of the ith paragraph after all the heads of the Inter-paragraph Attention mechanism are passed through: Among them, h represents the number of heads, It represents the context vector obtained after the i-th paragraph passes through the Inter-paragraphAttention mechanism of the z-th head. The T in represents the transpose operation. represents the learnable parameter matrix in the Inter-paragraph Attention mechanism of the z-th head, Represents vector query, key and value vectors, which are obtained from y ij Linear transformation.

8. The Transformer-based modality heterogeneous summary sentiment analysis system according to claim 5, characterized in that: The graph encoding layer in step S202 is composed of multiple graph encoding layers, and the graph encoding layer is implemented using technologies such as graph attention mechanism or graph neural network. The specific steps are as follows: Subgraph G of the undirected weighted graph i Input Graph Encoding Layer, perform graph encoding on each subgraph to obtain node features and edge features. Node features are represented by matrix V, and edge features are represented by matrix E. Each row of E i The feature vector representing the i-th edge is obtained by fusing node features and edge features into a graph representation vector, using an enhanced graph based on a graph attention network mechanism to calculate the correlation between nodes in the vector and update the feature vector of the node. That is, the explicit graph representation is input into the encoding process through the graph attention network mechanism, and the explicit relationship in the graph is used to learn better inter-sentence relationships. Each sentence collects information from other related sentences, thereby obtaining global information from the entire input. set up represents the output vector of the i-th sentence at the t-1th graph encoding layer, where is the input sentence vector; for each node i, the context representation u i is the vector representation of the ith node, obtained by the following steps: a ij =softmax(e ij +R ij ) Among them, α ij represents the weight coefficient between the i-th node and the j-th node; W K ,W Q ,W V is the parameter weight, which is used to calculate the parameter matrix of query, key and value respectively; e ij represents the potential relationship weight between the i-th node and the j-th node, d head represents the dimension of each attention head, R ij Represents the relative position information between the i-th node and the j-th node; Finally, u′ i After two layers of feedforward neural network, the feature vector u is output i .

9. The Transformer-based modality heterogeneous summary sentiment analysis system according to claim 8, characterized in that: R ij Determined by calculating the connection between two nodes in the undirected weighted graph G: Among them, σ represents the standard deviation of the strength of the graph structure influence; σ is a hyperparameter; G[i][j] is the element in the undirected weighted graph G.

10. The Transformer-based modality heterogeneous summary sentiment analysis system according to claim 8, characterized in that: The specific process of S203 includes the following steps: S2031, the feature matrix u i and i As the input of the encoder, a bidirectional LSTM or GRU is used to encode the feature matrix to obtain the hidden state matrix H, where each row represents the hidden state of a node or token; S2032. At the decoder end, a unidirectional LSTM or GRU is used to generate a summary sequence. The initial state is the last hidden state of the encoder, and the initial input is a special start symbol. <start> ;< / start> S2033, according to the output probability distribution p out , select the word or token with the highest probability as the output of the current decoding step; S2034, if the output word or token is a special end symbol <end> , then stop decoding, otherwise use the output word or token as the input of the next decoding step, and repeat the above operation until the maximum decoding length is reached or the end symbol is output;< / end> The summary sequence generated by the multimodal summary generation module is recorded as X α′ .

Citation Information

Patent Citations

  • BERT SUM and PGN fused supply chain ecological region long document abstract generation method

    CN115062140A

  • Model training method and device, electronic equipment and storage medium

    CN116070169A

  • Lightweight multi-modal sentiment analysis method

    CN117272226A

  • Cross-document false information detection method based on contrast graph learning

    CN117852526A

Cited By

  • Medical multi-mode hidden space alignment fusion method and system based on VQ-GCN

    CN119475243A

  • News customization-oriented multi-modal abstract generation method

    CN119621959A

  • Multi-modal common-situation analysis method and system, storage medium and electronic equipment

    CN120951275A

  • A multimodal co-emotion analysis method, system, storage medium and electronic device

    CN120951275B