Lightweight multimodal sentiment analysis method

Abstract

The application discloses a kind of lightweight multi-modal sentiment analysis methods.The sentiment analysis method includes: for the multi-modal sentiment data set collected in advance;Using multi-modal low-rank decomposition method, each modal feature sequence data is fused, and output fusion vector representation;Single cross-modal Transformer is used to carry out representation extraction processing to fusion vector representation and text feature sequence data, and obtain representation sequence;The representation sequence is input into a capsule graph network, and a graph vector is constructed;The graph node set in the graph vector and the adjacency matrix are aggregated;The graph structure representation output by each iteration is spliced, and the final output representation is obtained, which is used to judge the sentiment information.The sentiment analysis method of the application can effectively reduce the computing power consumption, improve the operation efficiency, and improve the performance of sentiment analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to a human emotion analysis technology, and in particular to a lightweight multi-modal emotion analysis method. BACKGROUND

[0002] The emotion analysis task is an emerging research field, and the main goal is to help computer systems to accurately identify, infer and interpret human emotions and opinions. This task plays a crucial role in human-computer interaction and can be applied to finance, education, medicine and other fields. As a multidisciplinary research field, research related to it has been widely concerned by domestic and foreign experts and scholars, and has become an important branch of deep learning research. Early emotion analysis methods mainly analyze features through single modal information such as voice, text or visual data to predict human emotions. However, in reality, there are many ways to express emotions, and people often express emotions through multiple ways at the same time. For example, when a person is saying something happy, they will also have a smile on their face. In addition, with the rapid development of social networks, various emotion-related data containing multi-modal information are also emerging. More and more researchers have begun to pay more attention to emotion analysis based on multi-modal data fusion.

[0003] The emotion analysis method based on multi-modal data fusion refers to the comprehensive analysis and prediction of human emotions by combining multiple modal data (such as voice, text, video, etc. mentioned above). This method generally first extracts single modal data through a single modal feature extractor, and then inputs it into a deep learning model for feature fusion and training, and finally realizes the function of emotion analysis. In comparison, this method can make up for the lack of information of single modal methods to some extent and provide reliable data input to improve the performance of emotion analysis. At present, many emotion analysis methods using multi-modal fusion have been proposed and have achieved good performance.

[0004] The current problems are in the following two aspects:

[0005] 1) Many existing methods often contain a large number of redundant network parameters in the process of feature extraction and model training, resulting in huge computational consumption and low efficiency of the model during operation. The main reason is that these methods do not consider how to simplify the model training and operation process, and ignore the imbalance of the contribution weight of different modal data to the performance of the model;

[0006] 2) With the emergence of various emotion data in social networks, the length of the emotion data sequence used for training is also increasing, and many emotion analysis methods have poor modeling ability for long-term dependence problems, which cannot accurately understand the emotions contained in long sequence data, thereby directly leading to unsatisfactory analysis performance of the model. SUMMARY

[0007] The purpose of the present application is to provide a lightweight multi-modal sentiment analysis method that can effectively reduce the consumption of computing power, improve the efficiency of operation, and improve the performance of sentiment analysis.

[0008] In order to achieve the above technical purpose, the present application adopts the following technical scheme:

[0009] A lightweight multi-modal sentiment analysis method, the sentiment analysis method comprising:

[0010] For a pre-collected multi-modal sentiment data set, the feature sequence data of the text, speech and video three modal data information of the sentiment data set is extracted;

[0011] A multi-modal low-rank decomposition method is used to fuse each modal feature sequence data, and output a fusion vector representation;

[0012] The fusion vector representation and the text feature sequence data are input into a single cross-modal Transformer, and a single cross-modal Transformer is used to perform representation extraction processing on the fusion vector representation and the text feature sequence data to obtain a representation sequence;

[0013] The representation sequence is input into a capsule graph network for graph construction, thereby constructing a graph vector composed of a graph node set and an adjacency matrix;

[0014] The graph node set and the adjacency matrix in the graph vector are subjected to an aggregation operation, and each iteration in the aggregation operation outputs a graph structure representation for sentiment analysis;

[0015] The graph structure representation output by each iteration is spliced to obtain a final output representation, which is used to make a judgment on the sentiment information.

[0016] Further, the multi-modal low-rank decomposition method used to fuse each modal feature sequence data includes the following specific implementation method:

[0017] A fixed rank r is pre-set, and for each weight tensor W in the weight tensor W k , the pre-set rank r is used to decompose the weight tensor W k , and the decomposition form is

[0018]

[0019] In the formula, the vector set is the r decomposition factor of the original tensor,

[0020] Then, the vector set reconstructing to obtain low-rank factors of M specific modalities,

[0021] for the vector set set the low-rank factor set in it as

[0022]

[0023] The calculation process of the weight tensor W is reconstructed as

[0024]

[0025] At the same time of completing the decomposition of the weight tensor W, the fusion tensor F is decomposed in parallel, and the calculation process of h M is converted to

[0026]

[0027] Then, the calculation of the fusion vector representation h M is obtained according to the converted calculation process of h M .

[0028] Further, the single cross-modal Transformer is used to perform representation extraction processing on the fusion vector representation and the text feature sequence data, and the specific implementation method includes:

[0029] The one-dimensional time convolution network is used to model the association relationship of each adjacent element in the fusion vector representation h M and the text feature sequence data x t , and the fusion vector representation h M and the text feature sequence data x t are projected to the same dimension to generate feature sequences and and are used to cooperate with the subsequent cross-modal Transformer to perform dot product operation,

[0030] and The calculation formula is as follows:

[0031]

[0032]

[0033] In the formula, Conv1D(·) is a one-dimensional time convolution operation, k f and k t are the convolution kernel size, T f and T t are the sequence length, and d is the pre-set projection dimension.

[0034] For characteristic sequences and Add location embeddings to obtain low-level location-aware features that can carry temporal information. and Specifically represented as

[0035]

[0036] In the formula, PE(·) represents the position embedding operation;

[0037] For low-level position-aware features and Will As a source mode sequence, The target modal sequence is input into a pre-defined cross-modal Transformer, which outputs the result of its last layer, i.e., the final text sequence representation after enhancement through multimodal data.

[0038] Using BiLSTM to represent text sequences The temporal information of each time step is collected and integrated to obtain a more robust representation sequence Z. f→t .

[0039] Furthermore, the specific method for inputting the representation sequence into a capsule graph network to construct a graph vector includes:

[0040] Capsule networks are used to construct the representation sequences into capsules;

[0041] A dynamic routing algorithm is used to transform the capsules into a set of graph nodes;

[0042] Based on the self-attention mechanism, an adjacency matrix is ​​constructed for a set of graph nodes.

[0043] Furthermore, the specific implementation method for the aggregation operation of the graph node set and adjacency matrix in the graph vector includes steps S51 to S53:

[0044] S51, for the graph node set and adjacency matrix

[0045] First, the graph node set is input into a pre-defined one-dimensional convolutional network to extract local features. Then, the local features are input into BiGRU, which further models the information of each time step in the local features based on the original capsule graph network, outputting a graph structure sequence. Finally, the graph structure sequence is multiplied with the adjacency matrix A and activated using the activation function tanh(·).

[0046] The mathematical expression is

[0047] n1=tanh(W0·BiGRU(Conv1D(N0))(A+I))

[0048] In the formula, N0 represents the set of graph nodes, A represents the adjacency matrix, W0 represents the weight, I represents the identity matrix, and I is used for self-loop operations;

[0049] After obtaining the output n1, an attention mechanism is introduced to enhance the model's learning and understanding of key information at each time step;

[0050] The mathematical expression for the introduced attention mechanism is:

[0051] N1 = Temporal_Attention(n1)

[0052] In the formula, N1 represents the set of graph nodes obtained in the first iteration, and Temporal_Attention(·) represents the attention mechanism;

[0053] S52, execute step S51 repeatedly for a pre-set number of iterations, and introduce max pooling operation between two iterations to alleviate overfitting and reduce the number of parameters;

[0054] The mathematical expression is

[0055] N i+1 =OuterLoop_Aggregation(MaxPooling(N i ))

[0056] In the formula, N i Let N represent the set of graph nodes obtained in the i-th iteration. i+1 This represents the set of graph nodes obtained in the (i+1)th iteration. MaxPooling(·) refers to the max pooling operation, and OuterLoop_Aggregation(·) refers to the outer loop graph aggregation.

[0057] S53, input the results of each outer loop graph aggregation iteration in step S52 into the capsule network, and use the dynamic routing algorithm to perform inner loop graph aggregation, finally obtaining the graph structure representation for sentiment analysis output by each iteration.

[0058] Furthermore, the extraction of feature sequence data from the three modalities of text, speech, and video in the sentiment dataset specifically involves using a single-modal feature extractor to extract the text, audio, and video data from the sentiment dataset.

[0059] The sentiment analysis method of this invention uses a fused vector representation as the source modality and text feature sequence data as the target modality during multimodal sentiment analysis. These are input into a single cross-modal Transformer for representation extraction. Compared to existing methods such as capsule graph networks, which directly use six cross-modal Transformers for representation extraction, this method uses only one cross-modal Transformer. Furthermore, the acquisition process of the fused vector representation utilizes a multimodal low-rank decomposition method for computational reconstruction and simplification, significantly reducing the number of network parameters required during model training. In addition, because the contribution weights of different modal data to model performance are unbalanced, this invention selects text feature sequence data with relatively high contribution weights and small data volume from each modality as the target modality. Through these two points, this invention can solve the problem mentioned in the background art of "existing methods consuming huge amounts of computational power and being inefficient in sentiment analysis" (Problem 1) with minimal performance loss; that is, it effectively reduces computational power consumption and improves computational efficiency.

[0060] This invention provides a sentiment analysis method that, based on the existing capsule graph network, improves the one-dimensional convolutional network used in the outer loop graph aggregation operation by replacing it with a CNN-BiGRU network with an attention mechanism. Since the attention mechanism and BiGRU network enhance the learning ability of temporal information in graph vectors, they improve the model's understanding of long-sequence input data containing sentiment information. Through this method, this invention solves the problem mentioned in the background art (Problem 2) that "existing methods have poor modeling ability for long-term dependency problems and unsatisfactory sentiment analysis performance," thus improving the performance of sentiment analysis. Attached Figure Description

[0061] Figure 1 This is a schematic diagram of the overall process of the lightweight multimodal sentiment analysis method of the present invention;

[0062] Figure 2 This is a diagram showing the overall network structure of the sentiment analysis method of the present invention.

[0063] Figure 3 This is a structural diagram of the cross-modal Transformer for the sentiment analysis method of the present invention;

[0064] Figure 4 This is the outer loop graph aggregation structure diagram of the sentiment analysis method of the present invention. Detailed Implementation

[0065] The present invention will be further illustrated below with specific embodiments:

[0066] This embodiment provides a lightweight multimodal sentiment analysis method that can effectively reduce computing power consumption, improve computational efficiency, and enhance sentiment analysis performance.

[0067] See Figure 1 The sentiment analysis method of this embodiment includes the following steps S1 to S6.

[0068] S1, for the pre-collected multimodal sentiment dataset, uses a single-modal feature extractor to extract feature sequence data from the text, speech and video data information in the sentiment dataset.

[0069] The “pre-collected emotion dataset” mentioned here comes from various publicly available datasets, mainly those that can reflect human emotional expression, including facial images, human voices, and various video clips with text annotations.

[0070] S2, the multimodal low-rank decomposition method is used to fuse the feature sequence data of each modality obtained in step 1, and the fused vector representation h is output. M .

[0071] It should be noted that the fusion vector representation h M The general calculation method is as follows

[0072] h M =W·F

[0073] in,

[0074] F represents the multimodal fusion tensor, which is based on It is determined.

[0075] W is a weight tensor of order (M+1), which is treated as... A tensor of order M, W k .

[0076] This conventional fusion vector representation h M The calculation method has the drawback of "a huge number of network parameters, which leads to low model training efficiency and high computing cost".

[0077] However, in this embodiment, a low-rank decomposition method is introduced to characterize the fusion vector h. M The computation process is reconstructed to avoid the unfavorable situation where "the number of parameters used for feature learning in the weight tensor W also grows exponentially due to the explicit creation of the fusion tensor F," thus making h... M The computational complexity is reduced significantly.

[0078] Specifically

[0079] See Figure 2In this embodiment, the multimodal low-rank decomposition method is used to fuse the feature sequence data of each modality obtained in step 1. The specific process includes steps S21 to S22.

[0080] S21, a fixed rank r is pre-defined, for each weight tensor W in the weight tensor W k The weight tensor W is subjected to a pre-defined rank r. k Decompose:

[0081] The decomposition form is

[0082]

[0083] In the formula,

[0084] Vector set That is, the r-factor of the original tensor.

[0085] Then, for the set of vectors Reorganize to obtain M low-rank factors for specific modes.

[0086] In this embodiment, for the vector set Let the set of low-rank factors be...

[0087]

[0088] Therefore, the calculation process of the weight tensor W can be reconstructed as follows:

[0089]

[0090] S22, while completing the decomposition of the weight tensor W in step S21, the fusion tensor F is decomposed in parallel, and h is... M The calculation process is transformed into:

[0091]

[0092] Then based on the transformed h M The calculation process is used to calculate the fusion vector representation h. M .

[0093] S3, a cross-modal Transformer is pre-set to process the "fusion vector representation h" obtained in step S2. M "As the source modality, and the text feature sequence data x obtained in step S1" t "As the target modality, it is input into the cross-modal Transformer, that is, a single cross-modal Transformer is used to process the fusion vector representation h." M "and text feature sequence data x"t "Perform representation extraction processing to obtain text sequence representations."

[0094] In addition, BiLSTM was used to collect and integrate the aforementioned text sequence representations. The temporal information in the sequence provides a more robust representation sequence Z for subsequent capsule graph networks. f→t .

[0095] Compared to existing capsule graph networks that use six cross-modal Transformers to extract representations, this implementation retains only one cross-modal Transformer to extract representations. Based on the imbalance of the contribution weights of different modal data to model performance, it selects text feature sequence data with relatively high contribution weights and small data volume in each modality as the target modality. This can significantly reduce the number of network parameters required for representation extraction with minimal loss in model performance.

[0096] The above-mentioned "using a single cross-modal Transformer to represent the fusion vector h" M "and text feature sequence data x" t The process of "character extraction processing" includes steps S31 to S34:

[0097] S31, using a one-dimensional temporal convolutional network to represent the fused vector h respectively. M and text feature sequence data x t The relationships between adjacent elements in the two sequences are modeled, and the fusion vector is used to represent h. M and text feature sequence data x t Projected onto the same dimension to generate feature sequences. and and It is used to perform dot product operations in conjunction with subsequent cross-modal Transformers.

[0098] and The calculation formula is as follows:

[0099]

[0100]

[0101] In the formula, Conv1D(·) is the one-dimensional temporal convolution operation, and k f k t T is the kernel size. f T t d is the sequence length, and d is the predefined projection dimension.

[0102] S32 is the feature sequence generated in step S31. and Add position embedding to obtain low-level position-aware features that can carry temporal information. and Specifically represented as

[0103]

[0104] In the formula, PE(·) is the position embedding operation.

[0105] S33, for the low-level position-aware features obtained in step S32 and Will As a source mode sequence, The target modal sequence is input into a pre-defined cross-modal Transformer, which outputs the result of its last layer, i.e., the final text sequence representation after enhancement through multimodal data.

[0106] The calculation process for layer 0 is as follows:

[0107]

[0108] like Figure 3 As shown, the calculation process of the i-th layer (i≥1) in one iteration is as follows:

[0109]

[0110]

[0111] Where LN(·) refers to layer normalization, f(·) refers to the position-wise feedforward fully connected layer, and CA(·) refers to the execution flow of the cross-modal Transformer, its general expression is:

[0112]

[0113] Where mode a is the target mode and b is the source mode.

[0114] It should be noted that the multimodal data refers to data derived from source modal sequences. Multimodal data in the data.

[0115] S34, Using BiLSTM to represent text sequences The temporal information at each time step is collected and integrated to provide a more robust representation sequence Z for subsequent capsule graph networks. f→t .

[0116] It should be noted that the "BiLSTM" is a bidirectional long short-term memory network, which is common knowledge to those skilled in the art.

[0117] S4, the representation sequence Z obtained in step S3 f→t Given a capsule graph network, the graph is first constructed, resulting in a graph vector composed of both the set of graph nodes and the adjacency matrix.

[0118] Specifically

[0119] S41, using capsule networks to represent sequence Z f→t Constructed as a capsule;

[0120] S42, using the existing dynamic routing algorithm, the capsule constructed in step S41 is transformed into a set of graph nodes, denoted by N0;

[0121] S43. Based on the self-attention mechanism of the existing technology, for the graph node set N0 obtained in step S42, construct an adjacency matrix that matches it, and denot the adjacency matrix A.

[0122] The graph node set constructed in step S42 and the adjacency matrix constructed in step S43 together form the graph vector.

[0123] S5. Perform an aggregation operation on the "graph node set N0" and "adjacency matrix A" obtained in step S4. Each iteration output during the aggregation operation is used for the graph structure representation of sentiment analysis.

[0124] The aggregation operations described here include two types: outer loop graph aggregation and inner loop graph aggregation. During each outer loop graph aggregation operation, all iterations of the inner loop graph aggregation operation are performed. The inner loop graph aggregation operation employs capsule networks and a dynamic routing algorithm.

[0125] It should be noted that the capsule graph network needs to be improved during the outer loop graph aggregation operation. This is mainly achieved by introducing a CNN-BiGRU network with an attention mechanism to enhance the learning and understanding of temporal information in the graph vectors, thereby further improving the model's ability to understand long-term dependencies based on the pre-defined capsule graph network. Specific improvements will be mentioned later.

[0126] Specifically, see Figure 4 The specific process of the above aggregation operation is as follows:

[0127] S51, for the "graph node set N0" and "adjacency matrix A" obtained in step S4

[0128] First, the graph node set N0 is input into a pre-defined one-dimensional convolutional network to extract local features. Then, the extracted local features are input into BiGRU, which further models the information of each time step in the local features based on the original capsule graph network, outputting a graph structure sequence. Finally, the graph structure sequence is multiplied with the adjacency matrix A and activated using the activation function tanh(·).

[0129] The above process can be expressed by mathematical expressions.

[0130] n1=tanh(W0·BiGRU(Conv1D(N0))(A+I))

[0131] Let W0 represent the weight, I represent the identity matrix, and I is used for the self-loop operation.

[0132] It should be noted that n1 here is an intermediate result in the calculation of N1, and it is also a graph structure sequence.

[0133] After obtaining the output n1 of the above equation, an attention mechanism is introduced to enhance the model's learning and understanding of key information at each time step, thereby further improving the model's ability to model long-term dependency problems based on the original capsule graph network. The mathematical expression for the introduced attention mechanism is:

[0134] N1 = Temporal_Attention(n1)

[0135] In the formula, N1 represents the set of graph nodes obtained in the first iteration, and Temporal_Aggregation(·) represents the attention mechanism.

[0136] S52, execute step S51 repeatedly for a pre-set number of iterations, and introduce max pooling between two iterations to alleviate overfitting and reduce the number of parameters.

[0137] The mathematical expression for this step is:

[0138] N i+1 =OuterLoop_Aggregation(MaxPooling(N i ))

[0139] In the formula, N i N represents the set of graph nodes obtained in the i-th iteration; i+1 This represents the set of graph nodes obtained in the (i+1)th iteration; MaxPooling(·) refers to the maximum pooling operation; OuterLoop_Aggregation(·) refers to the outer loop graph aggregation.

[0140] S53, input the results of each outer loop graph aggregation iteration in step S52 into the capsule network, and use the dynamic routing algorithm to perform inner loop graph aggregation, finally obtaining the graph structure representation for sentiment analysis output by each iteration.

[0141] S6. The graph structure representations of each iteration output in step S5 are concatenated to obtain the final output representation, which can be used to make judgments on the sentiment information contained in the original data of the input model.

[0142] Those skilled in the art will understand that the “model” described herein can be understood as the sentiment analysis method of this embodiment.

[0143] The main innovations of the sentiment analysis method in this embodiment lie in two aspects:

[0144] 1) The sentiment analysis method of this embodiment uses a fused vector representation as the source modality and text feature sequence data as the target modality when performing multimodal sentiment analysis. Both are input into a single cross-modal Transformer for representation extraction. Compared to existing methods such as capsule graph networks, which directly use six cross-modal Transformers for representation extraction, this embodiment uses only one cross-modal Transformer. Furthermore, the acquisition process of the fused vector representation utilizes a multimodal low-rank decomposition method for computational reconstruction and simplification, significantly reducing the number of network parameters required during model training. In addition, because the contribution weights of different modal data to model performance are unbalanced, this embodiment selects text feature sequence data with relatively high contribution weights and small data volume from each modality as the target modality. Through these two points, this embodiment can solve the problem mentioned in the background art of "existing methods consuming huge amounts of computational power and being inefficient when performing sentiment analysis" (Problem 1) with minimal performance loss.

[0145] 2) The sentiment analysis method in this embodiment improves upon the original capsule graph network by replacing the one-dimensional convolutional network used in the outer loop graph aggregation operation with a CNN-BiGRU network with an attention mechanism. Since the attention mechanism and BiGRU network enhance the learning ability of temporal information in graph vectors, they improve the model's understanding of long-sequence input data containing sentiment information. Through the above method, this embodiment can solve the problem mentioned in the background art that "existing methods have poor modeling ability for long-term dependency problems and unsatisfactory sentiment analysis performance" (Problem 2).

[0146] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A lightweight multimodal sentiment analysis method, characterized in that: The sentiment analysis method includes: For a pre-collected multimodal sentiment dataset, feature sequence data of text, speech and video data information are extracted from the sentiment dataset; A multimodal low-rank decomposition method is used to fuse the feature sequence data of each modality and output a fused vector representation. The fused vector representation and text feature sequence data are input into a single cross-modal Transformer. The single cross-modal Transformer is used to perform representation extraction processing on the fused vector representation and text feature sequence data to obtain the representation sequence. The representation sequence is input into a capsule graph network to construct a graph vector, which consists of a set of graph nodes and an adjacency matrix. An aggregation operation is performed on the set of graph nodes and the adjacency matrix in the graph vector. The output of each iteration in the aggregation operation is used for the graph structure representation of sentiment analysis. The graph structure representations output by each iteration are concatenated to obtain the final output representation, which is used to make judgments on sentiment information. The method for fusing feature sequence data from various modalities using a multimodal low-rank decomposition approach includes the following specific implementation methods: A fixed rank r is pre-defined for each weight tensor W. k The weight tensor W is subjected to a pre-defined rank r. k Decompose it into the following form: In the formula, the vector set That is, the r-factor of the original tensor. Then, for the set of vectors Recombination is performed to obtain M low-rank factors for specific modes. For the vector set Let the set of low-rank factors be... The calculation process of the weight tensor W is then reconstructed as follows: While performing the decomposition of the weight tensor W, the fusion tensor F is decomposed in parallel, and h is decomposed... M The calculation process is transformed into Then based on the transformed h M The calculation process is used to calculate the fusion vector representation h. M ; The method for using a single cross-modal Transformer to extract representations from fused vector representations and text feature sequence data includes: Using a one-dimensional temporal convolutional network to represent the fusion vector h M and text feature sequence data x t The relationships between adjacent elements in the two sequences are modeled, and the fusion vector is used to represent h. M and text feature sequence data x t Projected onto the same dimension to generate feature sequences. and and It is used to perform dot product operations in conjunction with subsequent cross-modal Transformers. and The calculation formula is as follows: In the formula, Conv1D(·) is a one-dimensional temporal convolution operation, and k f k t T is the kernel size. f T t d is the sequence length, and d is the predefined projection dimension; For characteristic sequences and Add location embeddings to obtain low-level location-aware features that can carry temporal information. and Specifically represented as In the formula, PE(·) represents the position embedding operation; For low-level position-aware features and Will As a source mode sequence, The target modal sequence is input into a pre-defined cross-modal Transformer, which outputs the result of its last layer, i.e., the final text sequence representation after enhancement through multimodal data. Using BiLSTM to represent text sequences The temporal information of each time step is collected and integrated to obtain a more robust representation sequence Z. f→t ; The method for inputting the representation sequence into a capsule graph network to construct a graph vector includes: Capsule networks are used to construct the representation sequences into capsules; A dynamic routing algorithm is used to transform the capsules into a set of graph nodes; Based on the self-attention mechanism, an adjacency matrix is ​​constructed for a set of graph nodes.

2. The lightweight multimodal sentiment analysis method according to claim 1, characterized in that: The method for performing the aggregation operation on the graph node set and adjacency matrix in the graph vector includes steps S51 to S53: S51, for the graph node set and adjacency matrix First, the graph node set is input into a pre-defined one-dimensional convolutional network to extract local features. Then, the local features are input into BiGRU, which further models the information of each time step in the local features based on the original capsule graph network, outputting a graph structure sequence. Finally, the graph structure sequence is multiplied with the adjacency matrix A and activated using the activation function tanh(·). The mathematical expression is n1 = tanh(W0·BiGRU(Conv1D(N0))(A+I)) In the formula, N0 represents the set of graph nodes, A represents the adjacency matrix, W0 represents the weight, I represents the identity matrix, and I is used for self-loop operations; After obtaining the output n1, an attention mechanism is introduced to enhance the model's learning and understanding of key information at each time step; The introduced attention mechanism is mathematically expressed as N1 = Temporal_Attention(n1). In the formula, N1 represents the set of graph nodes obtained in the first iteration, and Temporal_Attention(·) represents the attention mechanism; S52, execute step S51 repeatedly for a pre-set number of iterations, and introduce max pooling operation between two iterations to alleviate overfitting and reduce the number of parameters; The mathematical expression is N i+1 =OuterLoop_Aggregation(MaxPooling(N i )) In the formula, N i Let N represent the set of graph nodes obtained in the i-th iteration. i+1 This represents the set of graph nodes obtained in the (i+1)th iteration. MaxPooling(·) refers to the max pooling operation, and OuterLoop_Aggregation(·) refers to the outer loop graph aggregation. S53, input the results of each outer loop graph aggregation iteration in step S52 into the capsule network, and use the dynamic routing algorithm to perform inner loop graph aggregation, finally obtaining the graph structure representation for sentiment analysis output by each iteration.

3. The lightweight multimodal sentiment analysis method according to claim 1, characterized in that: The extraction of feature sequence data from the three modalities of text, speech, and video in the sentiment dataset specifically involves using a single-modal feature extractor to extract the text, audio, and video data from the sentiment dataset.

Images