Transform-based modal heterogeneous abstract sentiment analysis system

Abstract

The application discloses a modal heterogeneous abstract sentiment analysis system based on a transformer, and belongs to the technical field of information mining. In order to solve the problem that the traditional modal fusion method has insufficient complementary information mining, and further leads to the problem of poor sentiment analysis accuracy, the application comprises a pre-training feature extraction module, a multi-modal abstract generation module and a modal fusion module; wherein the pre-training feature extraction module comprises a text feature extraction unit for marking and coding text, and a speech feature extraction unit for discretizing and coding speech; the multi-modal abstract generation module firstly codes paragraphs and constructs an undirected weighted graph representing the similarity between sentences, clusters sub-graphs based on a graph convolution network, and codes sentences by using graph structure information; the feature vectors obtained by the graph coding layer and the paragraph coding layer are fused to generate an abstract sequence; and the modal fusion module performs sentiment classification based on the abstract text sequence and the speech feature.

Description

Technical Field

[0001] This invention belongs to the field of information mining technology and relates to a sentiment analysis system. Background Technology

[0002] With the rapid advancement of artificial intelligence technology, people's demands for intelligent human-computer interaction scenarios are constantly increasing. In real life, multimodal data typically consists of three modalities: visual (images), acoustic (speech), and transcribed text. Because the expression of human emotions is uncertain and diverse, multimodal sentiment analysis methods can combine complementary information from multiple sources such as text, speech, and images to provide more comprehensive and accurate emotion recognition results, enabling machines to understand human emotions in a way that is more in line with human habits. Therefore, modal heterogeneous summarization sentiment analysis methods have broad application prospects in many fields. However, current research on modal heterogeneous summarization sentiment analysis methods still faces some challenges, making it difficult to meet the needs of complex environments.

[0003] Traditional sentiment recognition methods often rely on low-level features, such as multi-layer multi-head fusion attention mechanisms and recurrent neural networks (RNNs) in speech emotion recognition. This leaves significant room for improvement and development in multimodal recognition capabilities. Existing multimodal sentiment analysis methods typically employ simple modality fusion approaches, such as early, mid, or late fusion. These methods ignore the heterogeneity and misalignment between different modalities, leading to the loss of complementary information and the introduction of noise, thus affecting the accuracy and robustness of sentiment analysis. Furthermore, while the Transformer has brought breakthroughs to generative text summarization, its ability to extract semantic relationships between sentences and paragraphs in the original text still needs improvement. Simple feature concatenation and fusion may result in significant data loss in some modalities, reducing the diversity between features and leading to lower recognition accuracy. Multimodal sentiment analysis methods often fail to consider the long textual information and multi-document information present in textual data. This information is crucial for understanding the main idea and sentiment of the text, but its complexity and diversity make it difficult to extract and utilize effectively.

[0004] For the reasons mentioned above, in an environment where the amount of network data is growing rapidly, it is crucial to provide an automatic, accurate, and multi-modal information-integrating summarization sentiment analysis method. Summary of the Invention

[0005] This invention aims to address the problem that traditional modality fusion-based sentiment analysis methods suffer from insufficient complementary information mining, which leads to poor sentiment analysis accuracy.

[0006] A Transformer-based modal heterogeneous summarization and sentiment analysis system includes a pre-trained feature extraction module, a multimodal summarization generation module, and a modality fusion module; among which,

[0007] The pre-trained feature extraction module includes a text feature extraction unit and a speech feature extraction unit;

[0008] Text feature extraction unit: Labels the text and encodes the labeled text to obtain X. α ; From feature information X a The document data is sent to the multimodal summary generation module;

[0009] Speech feature extraction unit: Discretizes the speech and encodes the discretized speech to obtain X. β ;

[0010] The multimodal summary generation module processes the following steps:

[0011] S201. Divide the document data set into paragraphs and input them into a paragraph encoding layer. The paragraph encoding layer uses Transformer to encode the paragraphs. Extract sentences from the output of the paragraph encoding layer and construct an undirected weighted graph G to represent the similarity between sentences, representing the similarity relationship between sentences in the summary text. Perform a clustering algorithm based on graph convolutional networks on graph G and divide graph G into several subgraphs, each subgraph representing a topic or a viewpoint.

[0012] S202. Input the subgraph of the undirected weighted graph into a graph coding layer and use the graph structure information to encode the sentence;

[0013] S203. The feature vectors obtained from the graph encoding layer and the segment encoding layer are fused, and a sequence with an attention mechanism and a pointer network is input into the sequence model to generate a summary sequence; the summary sequence generated by the multimodal summarization generation module is denoted as...

[0014] The processing procedure of the modal fusion module includes the following steps:

[0015] Summary text sequence for feature matrix representation And the X of voice β ,get and These correspond to the Query, Key, and Value matrices representing text and speech, respectively. These are the corresponding weight matrices;

[0016] Through multimodal fusion mechanism Get from β to α ′The cross-modal Attention output is represented as Right now:

[0017]

[0018] in, The scaling factor. and They have the same length;

[0019] Using the same approach, through a multimodal fusion mechanism Get from α ′ The cross-modal attention output to β is represented as

[0020] Use X con express Y β One of them, or Y β The fusion characteristics of X con After a one-dimensional convolutional layer, The data is then fed into an attention-based LSTM module; then into a fully connected layer and processed by softmax before the sentiment result is output.

[0021] Furthermore, the text feature extraction unit uses a GPT-2 tokenizer to tag the text, and uses a RoBERTa model to encode the tagged text to obtain X. α .

[0022] Furthermore, the speech feature extraction unit discretizes the speech using the VQ-Wav2Vec model and encodes the discretized speech using the Speech-BERT model to obtain X. β .

[0023] Furthermore, the segment encoding layer described in S201 consists of several Local Transformer Layers and Global Transformer Layers. The Local Transformer Layers and Global Transformer Layers are stacked alternately to form a deep Transformer network. Each Transformer Layer contains a Multi-Head Attention sublayer and a feedforward neural network sublayer. The Multi-Head Attention is a multi-head attention mechanism.

[0024] Furthermore, the specific process of S201 includes the following steps:

[0025] (1) Assume the document dataset contains n1 documents d1-d2. n1 Each document consists of m paragraphs; each document is segmented to obtain a set P = {P1, P2, P3, ..., P} of all paragraphs. m}, where P i This represents the i-th paragraph;

[0026] (2) For each paragraph P i The word segmentation process is performed to obtain a set W of all words. i ={w1,w2,…,w n1}, where w k This represents the j-th word;

[0027] (3) For each word w j The word vector e is obtained using a word embedding model. j Then, the position vector is obtained using a position encoding model. Adding the two together yields the word's representation vector x. j ,Right now

[0028] (4) Paragraph P i The representation vector x of all words in the text ij Form a matrix X∈R n*d , where n is the number of words in the paragraph, and d represents the dimension of the vector;

[0029] (5) The matrix X is used as the input of the paragraph encoding layer. First, it is input into the Local Transformer Layer and local self-attention encoding is performed on it by the Local Transformer Layer to obtain the encoding matrix Y of the paragraph.

[0030] (6) Input matrix Y into the Global Transformer Layer to obtain the context vector z for each paragraph. i ;

[0031] (7) For the context vector z of each paragraph i Input a two-layer feedforward network to obtain a summary vector s for each paragraph. i ;

[0032] (8) Summary vector s for each paragraph i Input a softmax layer to obtain the summary probability (prob) for each paragraph. i ∈[0,1], where prob i Indicates paragraph prob i The probability of being selected for inclusion in the abstract; specifically expressed as:

[0033]

[0034] Where, w∈R d represents the weight vector of the softmax layer, b∈R represents the bias scalar of the softmax layer, and m represents the number of paragraphs in the document;

[0035] (9) Based on the summarization probability of each paragraph (prob) i Select several paragraphs as candidate paragraphs for the summary, and then concatenate the candidate paragraphs in the order of the original document to form the paragraph summary text;

[0036] (10) Construct an undirected weighted graph from the output vector of the paragraph encoding layer, and calculate the cosine similarity matrix between sentences. The specific steps are as follows:

[0037] The encoding matrix of each paragraph in the abstract text is denoted as Y. i The encoding matrix Y for each paragraph i The sentence segmentation model is used to segment the text into sentences, resulting in a set of sentences S = {s1, s2, ..., s...} for each paragraph. k}, where s j This represents the j-th sentence, where k is the number of sentences in the paragraph;

[0038] For each sentence s j The sentence vector v is obtained using a sentence encoding model. j Then, the position vector is obtained using a position encoding model. Adding the two together yields the sentence representation vector xos. j ,Right now

[0039] xos, the representation vector of all sentences in all paragraphs j Form a matrix XOS;

[0040] xos for every two sentences j xos k Calculate the cosine similarity sim(xos) between them. j xos k The algorithm first determines whether two sentences are connected based on a threshold θ. Then, using sentences as nodes and sentence similarity as edge weights, it constructs an undirected weighted graph G = (V, E) to represent the sentence similarity relationship in the summary text.

[0041] A graph G is processed by a graph convolutional network-based clustering algorithm to dynamically calculate the attention weights between nodes and divide graph G into several subgraphs G1, G2, ..., G. l G i =(V iE i V i It is G i The set of nodes, E i It is G i The set of edges in the middle; each subgraph G i It represents a theme or a viewpoint.

[0042] Furthermore, the processing procedure of the Local Transformer Layer includes:

[0043] Based on the input matrix composed of word embedding vectors, the input tokens are embedded and positionally encoded to obtain the initial representation of each token. Multi-HeadAttention is applied to the representation of each token to obtain the attention score of each token to other tokens in the paragraph. Residual connections are made between the attention scores. Then, Layer Normalization is performed, and a feedforward neural network is applied to obtain the final representation of each token. Residual connections are made between the final representations. Finally, Layer Normalization is performed to obtain the output representation of each token.

[0044] Furthermore, the matrix Y is input into the Global Transformer Layer to obtain the context vector z for each paragraph. i The specific process is as follows:

[0045] z i =LayerNorm(g i +y ij )

[0046] Where z i ∈R d , z i It is the i-th row of Y; LayerNorm represents the layer normalization operation; g i Indicates header information c i With each input token vector y ij The vector obtained after addition is:

[0047]

[0048] in, and c represents the weight matrices of the first and second layers of the feedforward neural network, respectively. i This represents the sum of the context vectors obtained after the i-th paragraph has passed through the Inter-paragraph Attention mechanism of all headers:

[0049]

[0050] Where h represents the number of heads. This represents the context vector obtained after the i-th paragraph passes through the z-th header using the Inter-paragraph Attention mechanism. In this context, T represents the transpose operation. Let represent the learnable parameter matrix in the Inter-paragraph Attention mechanism of the z-th head. Representing a vector query, a key-value vector, is derived from y ij It comes from a linear transformation.

[0051] Furthermore, the graph encoding layer in step S202 consists of multiple graph encoding layers, which are implemented using techniques such as graph attention mechanisms or graph neural networks; the specific steps are as follows:

[0052] Subgraph G of the undirected weighted graph i The input is a Graph Encoding Layer, which encodes each subgraph to obtain node features and edge features. Node features are represented by matrix V, and edge features are represented by matrix E. Each row of E... i Let represent the feature vector of the i-th edge; fused the node features and edge features into a graph representation vector, and used an augmented graph based on a graph attention network mechanism to calculate the correlation between nodes in the vector and update the node feature vector; that is, the explicit graph representation input encoding process is carried out through a graph attention network mechanism, and the explicit relations in the graph are used to learn better inter-sentence relations; each sentence collects information from other related sentences, thereby obtaining global information from the entire input;

[0053] set up Let represent the output vector of the i-th sentence at the (t-1)-th graph coding layer, where It is the input sentence vector; for each node i, the context representation u i It is the vector representation of the i-th node, obtained through the following steps:

[0054] α ij =softmax(e ij +R ij )

[0055]

[0056] Where, α ij W represents the weight coefficient between the i-th node and the j-th node; K W Q WV These are the parameter weights, used to calculate the parameter matrices for the query, key, and value, respectively; e ij d represents the potential relationship weight between the i-th node and the j-th node. head R represents the dimension of each attention head. ij This represents the relative position information between the i-th node and the j-th node;

[0057] Finally, u′ i The feature vector u is output after two layers of feedforward neural network. i .

[0058] Furthermore, R ij This is determined by calculating the connectivity between two nodes in the undirected weighted graph G:

[0059]

[0060] Where σ represents the standard deviation of the influence of the graph structure on the strength; σ is a hyperparameter; G[i][j] are elements in the undirected weighted graph G.

[0061] Furthermore, the specific process of S203 includes the following steps:

[0062] S2031, the characteristic matrix u i and s i As input to the encoder, a bidirectional LSTM or GRU is used to encode the feature matrix to obtain the hidden state matrix H, where each row represents the hidden state of a node or token.

[0063] S2032. At the decoder end, a unidirectional LSTM or GRU is used to generate the digest sequence. The initial state is the last hidden state of the encoder, and the initial input is a special start symbol. <start>;

[0064] S2033, According to the output probability distribution p iut Select the word or token with the highest probability as the output of the current decoding step;

[0065] S2034. If the output word or token is a special end-of-line character. <end>If the decoding fails, the decoding stops; otherwise, the output word or token is used as the input for the next decoding step, and the above operation is repeated until the maximum decoding length is reached or the end symbol is output.

[0066] The summary sequence generated by the multimodal summary generation module is denoted as .

[0067] Beneficial effects:

[0068] This invention proposes a novel multimodal summarization sentiment analysis system based on Transformer and attention mechanisms. While effectively preserving low-level data features, it can extract features from multiple modalities, including text and speech. It then uses graph structure information and Transformer models to generate concise, accurate, and coherent summaries. Finally, it utilizes a bidirectional Transformer fusion network for modal fusion, which can fully mine the complementary information of different modalities, achieving efficient, accurate, and robust sentiment analysis. Attached Figure Description

[0069] Figure 1 This is a flowchart of the multimodal summarization sentiment analysis processing based on Transformer. Detailed Implementation

[0070] The present invention will now be described in conjunction with specific embodiments.

[0071] Specific implementation method one: Combining Figure 1 This implementation method is described below.

[0072] This implementation is a Transformer-based modal heterogeneous summarization and sentiment analysis system, including a pre-trained feature extraction module, a multimodal summarization generation module, and a modality fusion module; wherein,

[0073] The pre-trained feature extraction module extracts feature vectors from multiple modalities, including text and speech, and uses these vectors as input for the next module. This module employs a self-supervised learning method to learn general image features from large-scale unlabeled data and uses a pre-trained language model to learn semantic features from text data. This reduces the cost of collecting and annotating large-scale datasets and improves the generalization and transferability of feature representations.

[0074] Multimodal summarization generation module: This module is responsible for generating a summary sequence from the text feature vectors obtained from the pre-trained feature extraction module and using it as input to the next module. This module adopts a generative summarization method based on the Transformer model, using a document-level sentence relationship graph to capture long text information from multiple documents, and using a graph attention network mechanism to perform attention calculations on paragraphs and documents, optimizing the summary generation process and enabling the model to generate concise, coherent, and accurate summary text.

[0075] Modality Fusion Module: This module is responsible for performing sentiment analysis on the speech feature vectors obtained from the pre-trained feature extraction module and the summary sequences obtained from the multimodal summarization generation module, and outputting the sentiment category and confidence score. This module adopts a bidirectional Transformer fusion network, utilizing collaborative learning to build bidirectional interactions, achieving robustness to missing modalities, and improving the accuracy and robustness of sentiment analysis.

[0076] More specifically,

[0077] (I) Pre-trained feature extraction module:

[0078] Human language is a multimodal information carrier, encompassing not only natural language and text but also non-verbal modal information such as facial expressions and vocalizations. Therefore, in the data preprocessing stage, the pre-trained feature extraction module employs different methods to extract feature vectors from text and speech modalities. The pre-trained feature extraction module's processing includes text feature extraction units and speech feature extraction units:

[0079] For text data, the GPT-2 tokenizer is used to tag the text, and the RoBERTa model is used to encode the tagged text to obtain X. α Through continuous alternating training, GPT-2 can gradually improve the coherence, relevance and diversity of labeled text, while RoBERTa can gradually improve its ability to distinguish between real text and generated text.

[0080] For the speech data, the VQ-Wav2Vec model is used to discretize the speech, and the Speech-BERT model is used to encode the discretized speech to obtain X. β .

[0081] In practical use, speech signals and text are input into their respective pre-trained language models. Once the speech and text modalities are labeled, they are sent for subsequent processing; the feature information X... α The document data (processed by the pre-trained model RoBERTa) is sent to the multimodal summarization generation module.

[0082] (II) Multimodal summary generation module:

[0083] S201. The document dataset is segmented into paragraphs and input into a paragraph encoding layer. The paragraph encoding layer uses a Transformer to encode the paragraphs. Sentences are extracted from the output of the paragraph encoding layer, and an undirected weighted graph G representing the similarity between sentences is constructed to represent the similarity relationship between sentences in the summary text. Graph G is processed by a clustering algorithm based on graph convolutional networks, and graph G is divided into several subgraphs G1, G2, ..., G... l Each subgraph G i It represents a theme or a viewpoint.

[0084] The paragraph encoding layer consists of several Local Transformer Layers and Global Transformer Layers, which are alternately stacked to form a deep Transformer network. Each Transformer Layer contains a Multi-Head Attention sublayer and a feedforward neural network sublayer. The Multi-Head Attention sublayer allows the model to learn the dependencies between any two positions in the input sequence, while the feedforward neural network sublayer allows the model to learn the non-linear transformations at each position in the input sequence. The input and output of both the Multi-Head Attention sublayer and the feedforward neural network sublayer are matrices X∈R composed of word embedding vectors. n*d Each word embedding vector x ij The dimension is d model .

[0085] S202. Input the undirected weighted graph into a graph coding layer and use the graph structure information to encode the sentence;

[0086] S203. The feature vectors obtained from the graph coding layer and the paragraph coding layer are fused together and input into a sequence model with an attention mechanism and a pointer network to generate a summary sequence.

[0087] in,

[0088] The specific process of S201 is as follows:

[0089] (1) Assume the document dataset contains n1 documents d1-d2. n1 Each document consists of m paragraphs; each document is segmented to obtain a set P = {P1, P2, P3, ..., P} of all paragraphs. m }, where P i This represents the i-th paragraph;

[0090] (2) For each paragraph P i The word segmentation process is performed to obtain a set W of all words. i ={w1,w2,…,w n1 }, where w j This represents the j-th word;

[0091] (3) For each word w j The word vector e is obtained using a word embedding model. j Then, the position vector is obtained using a position encoding model. Adding the two together yields the word's representation vector x. j ,Right now

[0092] (4) Paragraph P i The representation vector x of all words in the text ij Form a matrix X∈R n*d , where n is the number of words in the paragraph, and d represents the dimension of the vector;

[0093] (5) The matrix X is used as the input of the paragraph encoding layer. First, it is input into the Local Transformer Layer and local self-attention encoding is performed on it to obtain the encoding matrix Y of the paragraph.

[0094] The Local Transformer Layer allows each token to focus on contextual information within the same paragraph. It consists of a Multi-Head Attention sublayer and a Feedforward Neural Network (FFN) sublayer. Multi-Head Attention, or MultiHead(Q,K,V), introduces a multi-head attention mechanism that allows each token to focus on other tokens with different attention distributions. In short, the Local Transformer Layer primarily encodes the contextual information of each token within a paragraph.

[0095] The input tokens are embedded and positionally encoded to obtain the initial representation of each token.

[0096] Multi-head attention is applied to the representation of each token to obtain the attention score of each token to other tokens in the paragraph; residual connections are then performed on the attention scores.

[0097] Then, layer normalization is performed, and a feed-forward network is applied to obtain the final representation of each token. Residual connections are then made on the final representation.

[0098] Finally, through layer normalization, the output representation of each token is obtained.

[0099] For the m-th Local Transformer Layer, the input is X (m-1) The output is X m , where X (m-1) and X m These are matrices composed of word embedding vectors, where each row represents a vector representation of a token, and the dimension of a token's embedding representation is d. model X (m-1) The embedding representation of a paragraph has a dimension of n*d. model , where n is the number of tokens in the paragraph.

[0100] The specific processing steps for each Local Transformer Layer are as follows:

[0101] 5.1 Regarding X (m-1) A linear transformation yields Q, K, and V, where Q, K, and V represent the query, key, and value vectors, respectively.

[0102] Applying linear transformations respectively, we obtain z groups of Q, K, and V, each with dimension d. model .

[0103] 5.2 Apply scaled dot product attention to each group of Q, K, and V to obtain z output vectors, each with dimension d. model The formula for calculating scaled dot product attention is as follows:

[0104]

[0105] Where Q, K, V are from X (m-1) It is derived from a linear transformation, where T represents the transpose operation, d represents the dimension of the word embedding vector, and softmax is a normalization function used to calculate the attention weight of each token to other tokens.

[0106] 5.3 The vector obtained after the i-th paragraph passes through the multi-head attention sub-layer of the z-th head is the output vector of each head. Calculation as follows

[0107]

[0108] Among them, W i Q W i K W i V W O It is a learnable parameter matrix.

[0109] Concatenate the output vectors of multiple heads

[0110] MultiHead(Q,K,V)=Concat(head1,…,head h W O

[0111] Where h is the number of heads.

[0112] 5.4 For each token in the m-th Local Transformer Layer, the calculation is as follows:

[0113]

[0114] The Feed-Forward Network is calculated as follows:

[0115] FFN(h ij ) = max(0, h ij W1+b1)W2+b2

[0116] The output vector of the Local Transformer Layer is:

[0117] y ij =LayerNorm(h ij +FFN(h ij ))

[0118] in, b2∈R d It is a learnable parameter matrix and bias vector, h ij ∈R d This represents the output vector after the MultiHead sublayer in the m-th Local Transformer Layer; `max` is an element-wise maximum function used to increase the non-linearity of the model. ff It refers to the output dimension of the first layer of a feedforward neural network. It is an adjustable hyperparameter used to control the complexity and expressive power of the model.

[0119] Where LayerNorm is the layer normalization function, defined as follows:

[0120]

[0121] Where, γ∈R d , β∈R d It is a learnable parameter vector, μ∈R d , σ∈R d It is the mean and standard deviation vector of X in the last dimension, used to eliminate offset and scaling between different layers.

[0122] The encoding vector y of all paragraphs ij The matrix Y∈R is formed m*d .

[0123] (6) Input matrix Y into the Global Transformer Layer to obtain the context vector z for each paragraph. i The specific process is as follows:

[0124] z i =LayerNorm(g i +y ij )

[0125] Where z i ∈R d , z i This is the i-th row of Y; LayerNorm represents the layer normalization operation, used to alleviate the vanishing or exploding gradient problem and improve the stability and convergence speed of the model. i Indicates header information c i With each input token vector y ij The vector obtained after addition is:

[0126]

[0127] in, and c represents the weight matrices of the first and second layers of the feedforward neural network, respectively. i Let represent the sum of the context vectors obtained by the i-th paragraph after passing through the Inter-paragraph Attention mechanism of all headers, i.e.:

[0128]

[0129] Where h represents the number of heads. This represents the context vector obtained after the i-th paragraph passes through the z-th header using the Inter-paragraph Attention mechanism. It is a weighted sum of the value vectors of all paragraphs, with the weights calculated by the dot product attention function between the query and key vectors. Specifically, it is expressed as follows:

[0130]

[0131] Where T represents the transpose operation. This represents the learnable parameter matrix in the Inter-paragraph Attention mechanism of the z-th head, used to map the input vector into query, key, and value vectors. This represents Query, Key, Value, i.e., a vector query, a key-value vector, which is derived from y ij It is derived from a linear transformation, and the specific calculation is as follows:

[0132]

[0133] y ij X represents the vector obtained after the i-th paragraph is processed by the Lobal Transformer Layer. ij The output vector of the last word, Let d represent the representation matrix of the j-th word in the i-th paragraph, where n is the number of words in the paragraph and d is the dimension of the word embedding vector.

[0134] (7) For the context vector z of each paragraph i Input a two-layer feedforward network to obtain a summary vector s for each paragraph. i ∈R t*d , where s i It is z i The nonlinear transformation, where t is the number of tokens in the paragraph;

[0135] s i =W2ReLU(W1z) i +b1)+b2

[0136] in, These represent the weight matrices of the first and second layers of the feedforward neural network, respectively. Let d represent the bias vectors of the first and second layers, respectively. ff represents the hidden layer dimension of the feedforward neural network, and ReLU represents the activation function of the linear rectified unit.

[0137] (8) Summary vector s for each paragraph i Input a softmax layer to obtain the summary probability (prob) for each paragraph. i ∈[0,1], where prob i Indicates paragraph prob i The probability of being selected for inclusion in the abstract; specifically expressed as:

[0138]

[0139] Where, w∈R d represents the weight vector of the softmax layer, b∈R represents the bias scalar of the softmax layer, and m represents the number of paragraphs in the document.

[0140] (9) Based on the summarization probability of each paragraph (prob) i Select several paragraphs as candidate paragraphs for the summary, and then concatenate the candidate paragraphs in the order of the original document to form the paragraph summary text.

[0141] (10) Construct an undirected weighted graph from the output vector of the paragraph encoding layer, and calculate the cosine similarity matrix between sentences. The specific steps are as follows:

[0142] The encoding matrix Y for each paragraph i The sentence segmentation model is used to segment the text into sentences, resulting in a set of sentences S = {s1, s2, ..., s...} for each paragraph. k }, where s j This represents the j-th sentence, where k is the number of sentences in the paragraph;

[0143] For each sentence s j The sentence vector v is obtained using a sentence encoding model. j Then, the position vector is obtained using a position encoding model. Adding the two together yields the sentence representation vector xos. j ,Right now

[0144] xos, the representation vector of all sentences in all paragraphs j Form a matrix XOS;

[0145] xos for every two sentences j xos k Calculate the cosine similarity sim(xos) between them. j xos k ), and determine whether two sentences are connected based on a threshold θ = 0.5, i.e., E = {(s j ,s k )|sim(s j ,s k If the similarity between sentences is greater than 0.5, then an undirected weighted graph G = (V, E) is constructed with sentences as nodes and sentence similarity as edge weights to represent the sentence similarity relationships in the summary text.

[0146] A graph G is processed by a graph convolutional network-based clustering algorithm to dynamically calculate the attention weights between nodes and divide graph G into several subgraphs G1, G2, ..., G. l G i =(V i E i V i It is G i The set of nodes, E i It is G i The set of edges in the middle. Each subgraph G i It represents a theme or a viewpoint.

[0147] The specific process of S202 is as follows:

[0148] An undirected weighted graph is input into a graph encoding layer. The sentence is encoded using graph structure information and an attention-based encoding mechanism. The graph encoding layer consists of multiple graph encoding layers. Graph encoding is a method that uses graph structure information to encode the input subgraph, implemented using techniques such as graph attention mechanisms or graph neural networks. The specific steps are as follows:

[0149] Subgraph G of the undirected weighted graph i The input graph encoding layer encodes each subgraph, yielding node and edge features. Node features are represented by a u×d1 matrix V, where n is the number of nodes in the graph and d1 is the dimension of the node features. Edge features are represented by an m×d2 matrix E, where m is the number of edges in the graph and d2 is the dimension of the edge features. Each row of E... i Let represent the feature vector of the i-th edge. Node features and edge features are fused into a graph representation vector. An augmented graph based on a graph attention network (Graph Attention Network) mechanism is used to calculate the correlations between nodes in the vector and update the node feature vectors. That is, the explicit graph representation input is encoded using a graph attention network mechanism, leveraging explicit relationships in the graph to learn better inter-sentence relationships. Each sentence can gather information from other related sentences, thus acquiring global information from the entire input. Let ... Let represent the output vector of the i-th sentence at the (t-1)-th graph coding layer, where It is the input sentence vector. For each node i, the context representation u i This refers to the vector representation of the i-th node (sentence), which integrates the relational information of other nodes (sentences) within that node (sentence). It is calculated as a weighted sum of the node (sentence) vectors after linear transformation, specifically as follows:

[0150] Assume the length of the input sequence is n s The dimension of the feature vector is If the number of heads is h, then the dimension of each head is...

[0151] α ij =softmax(e ij +R ij )

[0152]

[0153] In this context, the weighted sum refers to multiplying the value vector of each node (sentence) by a weight coefficient and then summing them to obtain the context representation. α ij W represents the weight coefficient between the i-th node (sentence) and the j-th node (sentence). K W Q W V ∈R d*d These are the parameter weights, used to calculate the parameter matrices for the query, key, and value, respectively. ij Let d represent the potential relation weight between the i-th node (sentence) and the j-th node (sentence). head R represents the dimension of each attention head. ij This represents the relative position information between the i-th node (sentence) and the j-th node (sentence). R ij It is obtained by calculating the connections between two nodes (sentences) in an undirected weighted graph G. The main difference in graph self-attention mechanisms is the additional pairwise bias R. ij It is calculated using the Gaussian bias of the weights of the undirected weighted graph G:

[0154]

[0155] Here, σ represents the standard deviation of the influence of the graphic structure on the strength of the effect. σ is a hyperparameter that can be adjusted based on experimental data to achieve the best results. A larger value of σ indicates a weaker influence from the graphic structure, and vice versa. The value of σ can also be determined through grid search or other optimization methods.

[0156] An undirected weighted graph G is a two-dimensional matrix that represents whether there are connections between different nodes (sentences) in the graph. If there is a connection between two nodes (sentences), the corresponding position is 1; otherwise, it is 0. G[i][h] represents the connection between the i-th node and the j-th node. Gaussian bias R ij ∈(-inf, 0) measures S i and S j The tightness between them, where the Gaussian bias R ij -inf is a negative number that measures the tightness between the i-th and j-th nodes. The closer -inf is to 0, the tighter the relationship between the two nodes; the closer -inf is to negative infinity, the looser the relationship between the two nodes. Due to the exponential operation in the softmax function, Gaussian bias is approximately equivalent to multiplying the latent attention distribution by the weights ∈ (0, 1).

[0157] u′ i The feature vector u is output after two layers of feedforward neural network. i ;

[0158] The specific process for S203 is as follows:

[0159] The feature vector u obtained from the graph coding layer and the segment coding layer i and s i The sequence is fused and used as input to a sequence-to-sequence model. This invention uses a sequence-to-sequence model with an attention mechanism and a pointer network to generate summary sequences. Specifically, the sentence vectors obtained from the graph encoding layer and the token vectors obtained from the paragraph encoding layer are concatenated according to the node's position in the graph, resulting in a feature matrix that fuses graph structure information and paragraph information. Details are as follows:

[0160] S2031, the characteristic matrix u i and s i As input to the encoder, a bidirectional LSTM or GRU is used to encode the feature matrix, resulting in an (n) n The hidden state matrix H is denoted by n(t)×2d, where each row represents the hidden state of a node (sentence) or token, and t is the number of tokens in the paragraph, n is the number of tokens in the paragraph, and n is the number of tokens in the paragraph. n d represents the number of nodes (sentences) in the graph. n It is the dimension of the node (sentence) features.

[0161] S2032. At the decoder end, a unidirectional LSTM or GRU is used to generate the digest sequence. The initial state is the last hidden state of the encoder, and the initial input is a special start symbol. <start>.

[0162] S2033, According to the output probability distribution p out The word or token with the highest probability is selected as the output of the current decoding step.

[0163] S2034. If the output word or token is a special end-of-line character. <end>If the decoding fails, the decoding stops; otherwise, the output word or token is used as the input for the next decoding step, and the above operation is repeated until the maximum decoding length is reached or the end symbol is output.

[0164] The summary sequence generated by the multimodal summary generation module is denoted as .

[0165] (III) Modal Fusion Module

[0166] To achieve bidirectional cross-modal feature information association, this invention employs a multimodal fusion mechanism, which can perform two modal α-processes in parallel. ′ The features of α and β are interacted and fused. Specifically, assuming α ′ Modality represents the summarized text processed by the multimodal summarization generation module, and β-modality represents speech. Let represent the input feature matrices for the abstract text and the speech, respectively. and T β These represent the number of tokens in the text and the time step of the audio, respectively. and d β Let represent the feature dimensions of text and speech, respectively. To enable cross-modal feature interaction and fusion, an attention mechanism (AM) is used, which computes the output attention based on the input queries, keys, and values. Specifically, let and The matrices representing the Query, Key, and Values ​​of text and speech respectively are as follows: and d represents the weight matrices for queries, keys, and values ​​in text and speech, respectively. k and d v These represent the feature dimensions of Keys and Values, respectively. A multimodal fusion mechanism is used. and We can obtain values ​​from β to α. ′ Cross-modal Attention output:

[0167] At this point, the output changes from β to α. ′ Cross-modal Attention:

[0168]

[0169] in, The scaling factor. and Having the same length (i.e.) );

[0170] From this, we can also obtain the result from α. ′ Cross-modal attention to β:

[0171]

[0172] Use X con express Y β One of them, or Y β The fusion features are obtained through a one-dimensional convolutional layer. Then it is input into the attention-based LSTM module.

[0173] The calculation process for one-dimensional convolutional layers is as follows:

[0174]

[0175] in, It is the size of the modal convolution kernel, d c It is a common dimension of cross-modality.

[0176] Mapping different modal features onto the same dimension makes it easier to further utilize the explicit interactions between modalities.

[0177] Finally, the data is fed into a fully connected layer and processed by softmax to output the sentiment results, thus achieving sentiment analysis of the summary.

[0178] This invention may have other embodiments. Without departing from the spirit and essence of this invention, those skilled in the art can make various corresponding changes and modifications according to this invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.< / end> < / start> < / end> < / start>

Claims

1. A modal heterogeneous summarization sentiment analysis system based on Transformer, characterized in that, It includes a pre-trained feature extraction module, a multimodal summarization generation module, and a modality fusion module; among which, The pre-trained feature extraction module includes a text feature extraction unit and a speech feature extraction unit; Text feature extraction unit: Labels the text and encodes the labeled text to obtain... ; from feature information The document data is sent to the multimodal summary generation module; Speech feature extraction unit: Discretizes the speech and encodes the discretized speech to obtain... ; The multimodal summary generation module processes the following steps: S201. The document data set is segmented into paragraphs and input into a paragraph encoding layer. The paragraph encoding layer uses a Transformer to encode the paragraphs. Sentences are extracted from the output of the paragraph encoding layer, and an undirected weighted graph G representing the similarity between sentences is constructed to represent the similarity relationship between sentences in the summary text. Graph G is processed by a graph convolutional network-based clustering algorithm, and graph G is divided into several subgraphs, each representing a topic or a viewpoint. This includes the following steps: (1) Assume that the document dataset contains n1 documents. - Each document consists of m paragraphs; each document is segmented to obtain a set P = { },in This represents the i-th paragraph; (2) For each paragraph Then, perform word segmentation to obtain a set of all words. ={ , ,…, },in This represents the j-th word; (3) For each word The word vectors are obtained using a word embedding model. Then, the position vector is obtained using a position encoding model. Adding the two together yields the word's representation vector. ,Right now = , (4) Paragraph Representation vectors of all words in the text Form a matrix ∈ , where n is the number of words in the paragraph, and d represents the dimension of the vector; (5) Transform the matrix As input to the paragraph encoding layer, it is first fed into the Local Transformer Layer, where it is locally self-attention encoded to obtain the paragraph encoding matrix Y. (6) Transform the matrix Input the Global Transformer Layer to obtain the context vector for each paragraph. ; (7) Context vector for each paragraph Input a two-layer feedforward network to obtain a summary vector for each paragraph. ; (8) Summary vector for each paragraph Input a softmax layer to obtain the summary probability of each paragraph. ∈[0,1], where Paragraph The probability of being selected for inclusion in the abstract; specifically expressed as: ,in, This represents the weight vector of the softmax layer. This represents the bias scalar of the softmax layer, where m represents the number of paragraphs in the document; (9) Based on the summarization probability of each paragraph Select several paragraphs as candidate paragraphs for the summary, and then concatenate the candidate paragraphs in the order of the original document to form the paragraph summary text; (10) Construct an undirected weighted graph from the output vectors of the paragraph encoding layer, and calculate the cosine similarity matrix between sentences. The specific steps are as follows: The encoding matrix of each paragraph in the abstract text is denoted as: The encoding matrix for each paragraph The sentence segmentation model is used to segment the text into sentences, resulting in a set of sentences for each paragraph. ,in Let j represent the j-th sentence, and k be the number of sentences in the paragraph; for each sentence The sentence vector is obtained using a sentence encoding model. Then, the position vector is obtained using a position encoding model. Adding the two together yields the sentence's representation vector. ,Right now = + ; Representation vectors of sentences in all paragraphs Form a matrix XOS; For every two sentences , Calculate the cosine similarity sim( between them). , ), and based on a threshold To determine whether two sentences are connected; then, using sentences as nodes and sentence similarity as edge weights, construct an undirected weighted graph G=(V,E) to represent the sentence similarity relationship in the summary text; A graph G is processed by a graph convolutional network-based clustering algorithm to dynamically calculate the attention weights between nodes and divide graph G into several subgraphs. , ,…, ,in , yes The set of nodes in the middle, yes The set of edges in the middle; each subgraph It represents a theme or a viewpoint; S202. Input the subgraph of the undirected weighted graph into a graph coding layer and use the graph structure information to encode the sentence; S203. The feature vectors obtained from the graph encoding layer and the segment encoding layer are fused, and a sequence with an attention mechanism and a pointer network is input into the sequence model to generate a summary sequence; the summary sequence generated by the multimodal summarization generation module is denoted as... ; The processing procedure of the modal fusion module includes the following steps: Summary text sequence for feature matrix representation and voice ,get , and These correspond to the Query, Key, and Value matrices representing text and speech, respectively. , These are the corresponding weight matrices; through a multimodal fusion mechanism , to obtain from β to The cross-modal Attention output is represented as ,Right now: ,in, The scaling factor. and They have the same length; Using the same approach, through a multimodal fusion mechanism , get from arrive The cross-modal Attention output is represented as ; use express , One of them, or , The fusion characteristics After a one-dimensional convolutional layer, The input is then fed into an attention-based LSTM module; then into a fully connected layer and passed through... The processed data outputs the sentiment results.

2. The modal heterogeneous summarization sentiment analysis system based on Transformer according to claim 1, characterized in that, The text feature extraction unit uses a GPT-2 tokenizer to tag the text and uses a RoBERTa model to encode the tagged text. .

3. The modal heterogeneous summarization sentiment analysis system based on Transformer according to claim 1, characterized in that, The speech feature extraction unit uses the VQ-Wav2Vec model to discretize the speech, and then uses the Speech-BERT model to encode the discretized speech. .

4. The modal heterogeneous summarization sentiment analysis system based on Transformer according to claim 1, characterized in that, The segment encoding layer described in S201 consists of several Local Transformer Layers and Global Transformer Layers. The Local Transformer Layers and Global Transformer Layers are stacked alternately to form a deep Transformer network. Each Transformer Layer contains a Multi-Head Attention sublayer and a feedforward neural network sublayer. The Multi-Head Attention is a multi-head attention mechanism.

5. The modal heterogeneous summarization sentiment analysis system based on Transformer according to claim 4, characterized in that, The processing procedure of the Local Transformer Layer includes: Based on the input matrix composed of word embedding vectors, the input tokens are embedded and positionally encoded to obtain the initial representation of each token. Multi-HeadAttention is applied to the representation of each token to obtain the attention score of each token to other tokens in the paragraph. Residual connections are made between the attention scores. Then, Layer Normalization is performed, and a feedforward neural network is applied to obtain the final representation of each token. Residual connections are made between the final representations. Finally, Layer Normalization is performed to obtain the output representation of each token.

6. The modal heterogeneous summarization sentiment analysis system based on Transformer according to claim 4, characterized in that, matrix Input the Global Transformer Layer to obtain the context vector for each paragraph. The specific process is as follows: in , This is the i-th row of Y; LayerNorm represents the layer normalization operation; Represents header information With each input token vector The vector obtained after addition is: in, and These represent the weight matrices of the first and second layers of the feedforward neural network, respectively. This represents the sum of the context vectors obtained after the i-th paragraph has passed through the Inter-paragraph Attention mechanism of all headers: Where h represents the number of heads. This represents the context vector obtained after the i-th paragraph passes through the Inter-paragraphAttention mechanism of the z-th header. In this context, T represents the transpose operation. Let represent the learnable parameter matrix in the Inter-paragraph Attention mechanism of the z-th head. , Representing vector queries, key-value vectors, is from... It comes from a linear transformation.

7. The modal heterogeneous summarization and sentiment analysis system based on Transformer according to claim 1, characterized in that, The graph encoding layer in step S202 consists of multiple graph encoding layers, which are implemented using graph attention mechanisms or graph neural network techniques; the specific steps are as follows: Subgraphs of an undirected weighted graph The input is a Graph Encoding Layer, which encodes each subgraph to obtain node features and edge features. The node features are represented by a matrix. The edge features are represented using a matrix. express, each line Indicates the first The feature vectors of each edge are fused into a graph representation vector. An augmented graph based on a graph attention network mechanism is used to calculate the correlation between nodes in the vector and update the feature vectors of the nodes. That is, the explicit graph representation input encoding process is carried out through a graph attention network mechanism, and the explicit relationships in the graph are used to learn better inter-sentence relationships. Each sentence collects information from other related sentences, thereby obtaining global information from the entire input. set up Let represent the output vector of the i-th sentence at the (t-1)-th graph coding layer, where It is the input sentence vector; for each node , context representation It is the first The vector representation of each node is obtained through the following steps: in, This represents the weight coefficient between the i-th node and the j-th node; , , These are parameter weights, used to calculate the parameter matrices for the query, key, and value, respectively. This represents the potential relationship weight between the i-th node and the j-th node. This represents the dimension of each attention head. This represents the relative position information between the i-th node and the j-th node; at last, The feature vector is output after two layers of feedforward neural network. .

8. The modal heterogeneous summarization sentiment analysis system based on Transformer according to claim 7, characterized in that, This is determined by calculating the connectivity between two nodes in the undirected weighted graph G: in, This represents the standard deviation of the strength influenced by the graphic structure; σ is a hyperparameter. Let G be an element in an undirected weighted graph.

9. The modal heterogeneous summarization sentiment analysis system based on Transformer according to claim 7, characterized in that, The specific process of S203 includes the following steps: S2031, the feature matrix and As input to the encoder, a bidirectional LSTM or GRU is used to encode the feature matrix to obtain the hidden state matrix H, where each row represents the hidden state of a node or token. S2032. At the decoder end, a unidirectional LSTM or GRU is used to generate the digest sequence. The initial state is the last hidden state of the encoder, and the initial input is a special start symbol. <start> ；< / start> S2033, Based on the output probability distribution Select the word or token with the highest probability as the output of the current decoding step; S2034. If the output word or token is a special end-of-line character. <end>If the decoding fails, the decoding stops; otherwise, the output word or token is used as the input for the next decoding step, and the above operation is repeated until the maximum decoding length is reached or the end symbol is output.< / end> The summary sequence generated by the multimodal summary generation module is denoted as . .

Images