Multi-modal sentiment analysis method based on gating circulation hierarchical fusion network

Abstract

The invention discloses a multi-modal sentiment analysis method based on a gating circulation hierarchical fusion network, which comprises the following steps: S1, multi-modal data preprocessing: collecting text, audio and video data, carrying out preprocessing and time sequence alignment, and constructing an annotation data set; s2, constructing a gating circulation hierarchical fusion network: designing a three-level network architecture comprising a modal feature extraction layer, a gating fusion layer and an emotion recognition layer; s3, model training and optimization: adopting an Adam optimizer, taking a cross entropy loss function as a target, carrying out iterative training on the labeled data set, and inhibiting overfitting through Dropout regularization and learning rate attenuation; s4, emotion analysis: inputting data acquired in real time into the trained model, analyzing the emotion state of the user in real time through a feature extraction layer, a gating fusion layer and an emotion recognition layer, and outputting an emotion analysis result; according to the method, information of different modes can be fully utilized in the sentiment analysis task, and the sentiment recognition accuracy is improved.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, specifically to a multimodal sentiment analysis method based on a gated recurrent hierarchical fusion network. Background Technology

[0002] Sentiment analysis is a core technology for achieving human-computer emotional interaction. Traditional methods often rely on a single text modality, ignoring emotional information in audio (such as tone and speed) and video (such as facial expressions and body language), leading to biased analysis results. Existing multimodal sentiment analysis models have the following limitations: The fusion strategy is simple: it adopts static fusion methods such as splicing or weighted summation, which cannot dynamically adapt to the emotional contribution of different modalities; Insufficient capture of temporal dependencies: The temporal characteristics of emotional expression are not fully utilized, such as the temporal correlation of tone shifts and facial expression changes; Redundant information interference: Multimodal data contains noise and repetitive features, which affect the accuracy of the model. Summary of the Invention

[0003] The purpose of this invention is to solve the above problems by designing a multimodal sentiment analysis method based on a gated recurrent hierarchical fusion network.

[0004] This invention provides a multimodal sentiment analysis method based on a gated recurrent hierarchical fusion network, the method comprising the following steps: S1. Multimodal data preprocessing: Collect text, audio, and video data, perform preprocessing and time-series alignment, and construct a labeled dataset; S2. Construction of Gated Recurrent Hierarchical Fusion Network: A three-level network architecture is designed, consisting of a modality feature extraction layer, a gated fusion layer, and an emotion recognition layer. The modality feature extraction layer extracts feature representations of different modal data, the gated fusion layer fuses multimodal features, and the emotion recognition layer uses the fused features to classify emotions. S3. Model Training and Optimization: The Adam optimizer is used, with the cross-entropy loss function as the target. Iterative training is performed on the labeled dataset, and overfitting is suppressed by Dropout regularization and learning rate decay. S4. Sentiment Analysis: Input the real-time collected data into the trained model, and analyze the user's emotional state in real time through the feature extraction layer, gating fusion layer and sentiment recognition layer, and output the sentiment analysis results.

[0005] Optionally, in the first implementation of the present invention, step S1 specifically includes the following steps: Text, audio, and video data are preprocessed separately to obtain text semantic features, acoustic features, and visual features; The DTW algorithm is used to match the timestamps of acoustic and visual features to the timeline of text semantic features; The preprocessed multimodal features are labeled to construct a labeled dataset, which is then divided into a training set, a validation set, and a test set.

[0006] Optionally, in a second implementation of the present invention, the preprocessing of text, audio, and video data to obtain text semantic features, acoustic features, and visual features includes: The text data is segmented using the BERT word segmenter, converted into word vectors, and stop words are removed. The hidden state output by the BERT word segmenter is used as the semantic features of the text. The wavelet thresholding method is used to filter noise in the audio data and extract acoustic features including Mel-frequency cepstral coefficients and fundamental frequency. Facial key points were extracted using OpenCV, and frame-level visual features were extracted using the ResNet50 model.

[0007] Optionally, in a third implementation of the present invention, the modal feature extraction layer includes a text feature extraction subnetwork, an audio feature extraction subnetwork, and a video feature extraction subnetwork, wherein: The text feature extraction subnetwork captures the sentiment tendency in the text based on the text's semantic features, and obtains the text feature vector; The audio feature extraction subnetwork uses 1D convolutional layers based on acoustic features to extract local temporal features in order to capture emotional cues and obtain audio feature vectors; The video feature extraction subnetwork reduces the dimensionality of visual features to 128 through pooling layers, capturing visual emotional signals including facial expressions and body movements to obtain video feature vectors.

[0008] Optionally, in a fourth implementation of the present invention, the gated fusion layer includes a Bi-GRU structure and an adaptive gating mechanism.

[0009] Optionally, in a fifth implementation of the present invention, the gated fusion layer employs a layered fusion strategy, which includes the following steps: During feature-level fusion, text feature vectors, audio feature vectors, and video feature vectors are concatenated and dimension-mapped to generate cross-modal feature vectors. When performing temporal hierarchical fusion, the Bi-GRU structure is used to perform temporal modeling of cross-modal feature vectors, output the temporal feature sequence of each modality, capture the temporal correlation of emotional expression, and dynamically adjust the contribution of each modality feature through gating weights; When performing decision-level fusion, global max pooling is applied to the time-series feature sequences to generate fused features.

[0010] Optionally, in the sixth implementation of the present invention, the emotion recognition layer uses a fully connected layer to perform dimensionality reduction and nonlinear transformation on the fused features, outputs the emotion classification result through the softmax activation function, and predicts the emotion intensity through a regression layer, wherein the emotion classification result includes positive, negative and neutral.

[0011] Optionally, in the seventh implementation of the present invention, step S3 specifically includes the following steps: Set the loss function, optimizer, and hyperparameters for model training. Use the training set as input to iteratively update the model parameters. After each training round, calculate the accuracy, F1 score, and intensity prediction error on the validation set. If the F1 score on the validation set does not improve for five consecutive epochs, training is terminated and the current optimal model parameters are saved.

[0012] Optionally, in the eighth implementation of the present invention, the training sample weights of the corresponding categories are increased for the emotion categories that perform poorly in the validation set; the GRU hidden dimension of the gated fusion layer is adjusted to enhance the ability to capture temporal dependencies; and data augmentation processing is introduced to enhance the robustness of the model to noisy data, wherein the data augmentation processing includes at least audio noise addition and random cropping of video frames.

[0013] Optionally, in the ninth implementation of the present invention, step S4 specifically includes the following steps: Multimodal data is acquired in real time. After preprocessing the acquired real-time data, multimodal feature sequences are obtained and input into the model. The modal feature extraction layer performs in-depth extraction of text semantic features, audio acoustic features, and video visual features to enhance the emotional cues of each modality; The gated fusion layer captures the temporal dependencies of each modality feature through a bidirectional GRU and uses a genetic algorithm to optimize and obtain the modality weight coefficients. The genetic algorithm encodes each modality weight coefficient as a chromosome and randomly generates an initial population containing several weight combinations within a preset range. The selection, crossover, and mutation processes are carried out until the optimal weight combination is found. The optimal weight combination output by the genetic algorithm is used as the adjustment range constraint of the adaptive gating mechanism. Then, the weights of each modality are dynamically adjusted through the adaptive gating mechanism to generate fusion features. The emotion recognition layer processes the fused features and outputs the emotion classification result, which completes the real-time analysis of the current user's emotional state. The model's output sentiment classification results are converted into natural language descriptions, transmitted to real-world scenarios, and user evaluations of the sentiment classification results are collected. When user labeling is inaccurate, record the corresponding multimodal raw data model analysis results and the user's true sentiment to establish a user feedback database. The user feedback database is added to the labeled dataset to form an incremental training dataset. The model is then adjusted and continuously updated using the incremental training dataset.

[0014] The technical solution provided by this invention involves collecting text, audio, and video data, performing preprocessing and time-series alignment to construct a labeled dataset; designing a three-level network architecture including a modality feature extraction layer, a gated fusion layer, and an emotion recognition layer; employing the Adam optimizer with cross-entropy loss as the target, iteratively training on the labeled dataset, and suppressing overfitting through Dropout regularization and learning rate decay; inputting real-time collected data into the trained model, and analyzing the user's emotional state in real time through the feature extraction layer, gated fusion layer, and emotion recognition layer, outputting emotion analysis results; the adaptive gating mechanism of this invention can adjust modality weights in real time, such as prioritizing the retention of audio and video features in anger, improving adaptability to complex scenarios; bidirectional GRU captures the temporal correlation of emotional expression, which is more in line with the laws of human emotional expression than static fusion; improving accuracy through three-level fusion at the feature level, time series level, and decision level; improving the emotional adaptability of service responses by analyzing user voice tone and facial expressions, and identifying multimodal signals of depressive tendencies in real time when used for mental health monitoring; and fully utilizing information from different modalities in emotion analysis tasks to improve the accuracy of emotion recognition. Attached Figure Description

[0015] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0016] Figure 1 This is a schematic diagram of the first embodiment of the multimodal sentiment analysis method based on a gated recurrent hierarchical fusion network provided by the present invention; Figure 2 This is a schematic diagram of a second embodiment of the multimodal sentiment analysis method based on a gated recurrent hierarchical fusion network provided in this invention. Detailed Implementation

[0017] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0018] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 A schematic diagram of the first embodiment of the multimodal sentiment analysis method based on gated recurrent hierarchical fusion network provided by this invention: S1. Multimodal data preprocessing: Collect text, audio, and video data, perform preprocessing and time-series alignment, and construct a labeled dataset; S2. Construction of Gated Recurrent Hierarchical Fusion Network: A three-level network architecture is designed, consisting of a modality feature extraction layer, a gated fusion layer, and an emotion recognition layer. In this embodiment, the modal feature extraction layer extracts feature representations of different modal data, the gated fusion layer fuses multimodal features, and the emotion recognition layer uses the fused features to classify emotions. In this embodiment, the modal feature extraction layer includes a text feature extraction subnetwork, an audio feature extraction subnetwork, and a video feature extraction subnetwork. Specifically: the text feature extraction subnetwork captures the emotional tendency in the text based on the text semantic features to obtain a text feature vector; the audio feature extraction subnetwork uses 1D convolutional layers based on acoustic features to extract local temporal features to capture emotional cues and obtain an audio feature vector; and the video feature extraction subnetwork reduces the dimensionality of visual features to 128 through pooling layers to capture visual emotional signals including facial expressions and body movements and obtain a video feature vector.

[0019] In this embodiment, the gating fusion layer includes a Bi-GRU structure and an adaptive gating mechanism.

[0020] In this embodiment, the gating fusion layer adopts a layered fusion strategy for fusion, which includes the following steps: During feature-level fusion, text feature vectors, audio feature vectors, and video feature vectors are concatenated and dimension-mapped to generate cross-modal feature vectors. When performing temporal hierarchical fusion, the Bi-GRU structure is used to perform temporal modeling of cross-modal feature vectors, output the temporal feature sequence of each modality, capture the temporal correlation of emotional expression, and dynamically adjust the contribution of each modality feature through gating weights; When performing decision-level fusion, global max pooling is applied to the time-series feature sequences to generate fused features.

[0021] In this embodiment, the emotion recognition layer uses a fully connected layer to perform dimensionality reduction and nonlinear transformation on the fused features, outputs the emotion classification result through the softmax activation function, and predicts the emotion intensity through a regression layer. The emotion classification result includes positive, negative, and neutral.

[0022] S3. Model Training and Optimization: The Adam optimizer is used, with the cross-entropy loss function as the target. Iterative training is performed on the labeled dataset, and overfitting is suppressed by Dropout regularization and learning rate decay. In this embodiment, combining the dual objectives of classification and intensity prediction in sentiment analysis, a combined loss function is adopted: the cross-entropy loss function is used to handle the sentiment classification task, such as positive, negative, and neutral categories, to measure the difference between the predicted category and the true category; the mean squared error loss function is used to handle the sentiment intensity prediction task, to measure the deviation between the predicted intensity and the labeled intensity; at the same time, an L2 regularization term is added to suppress overfitting by limiting the size of the model parameters. The final loss function is a weighted sum of the classification loss, the intensity prediction loss, and the regularization term. The Adam optimizer is used for iterative updates of model parameters. This optimizer combines the advantages of momentum gradient descent and adaptive learning rate, effectively accelerating convergence and improving training stability. The initial learning rate is set to 0.001, and momentum parameters β1 and β2 are set to 0.9 and 0.999 respectively to accommodate the complex parameter update requirements during multimodal feature fusion. The training batch size is set to 32, meaning 32 multimodal samples are input for parameter updates in each iteration, balancing computational efficiency and model convergence. The total number of training epochs is set to 50 to ensure the model has sufficient iterations to learn data patterns. Dropout regularization is introduced, with a Dropout rate of 0.3 set in the gated fusion layer and the sentiment recognition layer, randomly dropping some neuron connections to reduce the risk of overfitting. The preprocessed multimodal training data is read in batches, with each batch containing 32 samples. The features of each sample have been time-aligned to ensure consistent feature sequence lengths in the input model. The batch data is then input into a gated recurrent hierarchical fusion network, which sequentially passes through a modality feature extraction layer, a gated fusion layer, and a sentiment recognition layer. The network outputs a sentiment classification probability distribution and a predicted sentiment intensity value, while simultaneously recording the intermediate features and parameter states of each layer. Based on the forward propagation output, the loss value of the current batch is calculated using a pre-defined loss function. The gradient of each parameter is calculated layer by layer from the output layer to the input layer using the backpropagation algorithm to determine the direction and magnitude of parameter updates. Based on the adaptive learning rate mechanism of the Adam optimizer, the model parameters are iteratively updated using the calculated gradient information to complete the training process for one batch. This process continues until all samples in the training set have been traversed, completing one epoch of training. After each epoch of training, the model is switched from training mode to evaluation mode, and regularization operations such as Dropout are disabled to ensure that model parameters remain constant during validation and that evaluation results are stable and reliable. Preprocessed validation set samples are input into the model in batches, and sentiment classification predictions and intensity predictions are obtained through forward propagation without backpropagation or parameter updates. For sentiment classification results, the proportion of correctly predicted samples to the total number of samples is calculated to obtain the accuracy. Precision and recall are calculated separately for each sentiment category, and the F1 score is calculated using macro-averaging to comprehensively evaluate classification performance. For sentiment intensity prediction results, the mean absolute error and root mean square error (RMSE) between the predicted and true values ​​are calculated. The mean absolute error reflects the average deviation between the predicted and true values, while the RMSE is more sensitive to larger errors; both together measure the accuracy of intensity prediction. The accuracy, F1 score, mean absolute error, and RMSE of the current epoch are recorded as evaluation criteria for model performance. During training, a list recording the F1 scores of the validation set for each epoch is maintained. After each training round, the current F1 score is stored in the list, along with the corresponding model parameter file path and training epoch. The F1 score of the current epoch is compared with the F1 scores of the previous four epochs. If the F1 score does not improve for five consecutive epochs (i.e., the current F1 score is not higher than the maximum value among the previous four epochs), an early stopping condition is triggered, indicating that the model has converged or is showing an overfitting trend, and training does not need to continue. When the early stopping condition is met, training for subsequent epochs is stopped. The model parameter file corresponding to the epoch with the highest F1 score is found from the recorded list. This parameter file is the optimal model parameter during training. The optimal model parameter is saved locally in binary file form, including the weights, biases, and various hyperparameter configurations of the feature extraction layer, gated fusion layer, and sentiment recognition layer, ensuring the integrity and reusability of the parameters.

[0023] In this embodiment, the training sample weights of the corresponding categories are increased for the emotion categories that perform poorly in the validation set; the GRU hidden dimension of the gated fusion layer is adjusted to enhance the ability to capture temporal dependencies; and data augmentation processing is introduced to enhance the robustness of the model to noisy data, wherein the data augmentation processing includes at least audio noise addition and random cropping of video frames.

[0024] S4. Sentiment Analysis: Input the real-time collected data into the trained model, and analyze the user's emotional state in real time through the feature extraction layer, gating fusion layer and sentiment recognition layer, and output the sentiment analysis results.

[0025] In this embodiment, multimodal data is acquired in real time. After preprocessing the acquired real-time data, multimodal feature sequences are obtained and input into the model. The modal feature extraction layer performs deep extraction of text semantic features, audio acoustic features, and video visual features to enhance the emotional cues of each modality. The gated fusion layer captures the temporal dependencies of each modality feature through a bidirectional GRU, optimizes the modality weight coefficients using a genetic algorithm, uses the modality weight coefficients as the adjustment range constraints of the adaptive gating mechanism, and then dynamically adjusts the modality weights through the adaptive gating mechanism to generate fused features. The emotion recognition layer processes the fused features and outputs emotion classification results, which complete the real-time analysis of the current user's emotional state. The emotion classification results output by the model are converted into natural language descriptions, transmitted to the actual scene, and user evaluations of the emotion classification results are collected. When the user labeling is inaccurate, the corresponding multimodal raw data model analysis results and the user's labeled true emotions are recorded to establish a user feedback database. The user feedback database is supplemented into the labeled dataset to form an incremental training dataset. The incremental training dataset is used to adjust the model and continuously update the model.

[0026] In this embodiment, in the gated fusion layer, a bidirectional GRU is first used to perform temporal modeling on the features of text, audio, and video modalities, extracting forward and backward hidden state sequences to capture long-short-term dependencies. Subsequently, pre-optimized modal weight coefficients from the genetic algorithm, such as 0.55 for text, 0.25 for audio, and 0.20 for video, are input as initial parameters to the adaptive gating mechanism, forming a dynamically adjusted constraint range, such as text weight ∈ [0.45, 0.65]. Based on the feature distribution of the current input sample, the gating mechanism calculates the importance score for each modality: for the text modality, the emotional intensity of keywords is analyzed using an attention weight matrix; for the audio modality, emotionally sensitive features such as pitch and volume change rate are detected; for the video modality, temporal changes in facial micro-expressions and body movements are extracted. These scores are mapped to dynamic weights in the [0,1] interval using a sigmoid function. After intersecting with the constraint range of the genetic algorithm, the modal features output by the bidirectional GRU are weighted and fused to generate the final multimodal emotional feature representation. The genetic algorithm encodes the weight coefficients of each modality, such as text, audio, and video weights, into chromosomes, and randomly generates an initial population containing several sets of weight combinations within a preset range. Then, the fitness of each individual is evaluated by the model's sentiment analysis performance on the validation set, such as F1 score and accuracy, and individuals with high fitness are selected as parents. Through crossover operations, such as exchanging the weight values ​​of some modalities, and mutation operations, such as slightly and randomly adjusting the weight of a certain modality, offspring are generated. The above selection, crossover, and mutation processes are iterated until the optimal weight combination is evolved, which is used as an upper and lower bound constraint when adjusting the weights of each modality in an adaptive gating mechanism.

[0027] In this embodiment, the weights output by the genetic algorithm serve as the constraint range of the adaptive gating mechanism, dynamically adjusting the contribution of each modality. The modal weights optimized by the genetic algorithm, such as 0.55 for text, 0.25 for audio, and 0.20 for video, are used as initial weights. Based on the output of the genetic algorithm, the adjustment range of each modal weight is set, such as text weight ∈ [0.45, 0.65]. The weights are mapped to the [0, 1] interval through the sigmoid function.

[0028] In this embodiment, the GRU gating mechanism controls the flow of temporal information through update gates and reset gates to capture the temporal dependencies of multimodal features; the adaptive gating mechanism dynamically adjusts the modal weights according to the feature distribution of the input samples to solve the problem of redundant information interference; the GRU gating mechanism is responsible for temporal modeling, and the adaptive gating mechanism is responsible for the dynamic adjustment of modal weights. The combination of the two achieves the dual objectives of temporal dependency capture and modal contribution optimization.

[0029] In this embodiment, the adaptive gating mechanism calculates the importance score of each modality, inputs the modality importance score into the sigmoid function to generate dynamic weights, and combines the constraints of the genetic algorithm to obtain the final weights.

[0030] In this embodiment, the trained model is deployed in the following scenarios to verify its practical application effect: Intelligent customer service system: Real-time collection of users' text inquiries, voice tone, and facial expressions, outputting users' emotional states such as dissatisfaction or satisfaction, driving adjustments to customer service response strategies, such as transferring to human customer service when dissatisfied; Emotional robots: In the fields of education and mental health, they can recognize emotional changes such as frustration and happiness through multimodal input and adjust the interactive content accordingly, such as playing encouraging voice messages when frustrated. Social media analytics: Perform multimodal sentiment analysis on text comments, voice narration, and video footage on short video platforms to generate user sentiment trend reports, such as 80% of users expressing a positive attitude under a certain topic; Accuracy and Macro-F1 score are used to evaluate the model for classification tasks, while mean absolute error and root mean square error are used to evaluate the model's prediction strength. Collect error cases from real-world applications, such as video blurring leading to misjudgment of emotion, construct an incremental dataset, and fine-tune the model based on the incremental dataset, such as freezing the feature extraction layer and only updating the parameters of the gating fusion layer and the recognition layer; optimize the weight calculation logic of the gating mechanism based on user feedback, such as misjudgment cases marked by customer service personnel, such as enhancing the weight of audio features in low-quality video scenarios, and continuously improve the model's practical adaptability.

[0031] In this embodiment, affective computing, an important branch of artificial intelligence, focuses on enabling machines to understand and express emotions. Research on gated recurrent hierarchical fusion networks based on multimodal affective analysis can drive the development of affective computing in theory, technology, and application, promoting in-depth research and widespread application. Traditional affective analysis mainly relies on single-modal data, which limits the completeness of emotional expression and the accuracy of analysis. Multimodal affective analysis can fuse data from multiple modalities, thereby improving the accuracy and efficiency of affective analysis. Simultaneously, the design of gated recurrent hierarchical fusion networks can better handle sequential data, further enhancing the performance of affective analysis. Affective analysis has wide applications in many fields, such as social media analysis, human-computer interaction, intelligent customer service, and affective robots. Research and application of gated recurrent hierarchical fusion networks based on multimodal affective analysis can broaden the application areas of affective analysis, providing affective analysis support for more scenarios. Affective computing technology based on multimodal affective analysis can make human-computer interaction more natural and fluid, improving user experience. For example, intelligent assistants can understand users' emotions, thus providing services more intelligently; educational robots can adjust their teaching methods based on students' emotional feedback, etc.

[0032] In this embodiment, the key issues to be addressed include: 1) Effective fusion of multimodal data A key issue is how to effectively fuse data from different modalities so that the information from various modalities can complement each other and work together for sentiment analysis. It is necessary to design appropriate fusion strategies so that multimodal data can work together to improve the accuracy of sentiment recognition. 2) Optimization of gated loop networks Gated recurrent networks (GRNs) have significant advantages in processing sequential data, but they may face some challenges in multimodal sentiment analysis. How to optimize the structure and parameters of GRNs for the characteristics of multimodal data so that they can better handle multimodal sequential data is a key problem that needs to be solved. 3) Effectiveness of feature extraction and representation learning In multimodal sentiment analysis, extracting effective sentiment features from various modalities and learning representations of these features is crucial for improving the accuracy of sentiment analysis. It is necessary to study effective feature extraction methods and representation learning techniques to capture key sentiment information from multimodal data. 4) Design of layered fusion strategy Layered fusion strategy is an important aspect of multimodal sentiment analysis, which involves how to fuse information from different modalities at different levels. How to design a reasonable layered fusion strategy so that information from each level can be effectively fused and interact is a key problem that needs to be solved.

[0033] In this embodiment, 1) Data collection and processing CMU-MOSI, or Multimodal Sentiment Intensity Corpus dataset, contains 2199 video clips and is commonly used for multimodal sentiment analysis benchmarks. The video sentiment intensity is labeled in the range of [-3, 3] and is strictly labeled using subjective, sentiment intensity, visual, and audio feature labels. CMU-MOSEI, the largest multimodal sentiment analysis and recognition dataset, is an improved version of CMU-MOSEI; it contains more than 23,500 videos from more than 1,000 YouTube speakers; the dataset is gender-balanced, sentences come from different topics and monologue videos, and have been transcribed and punctuated; 2) Network Architecture Design: A gated recurrent hierarchical fusion network is designed, taking text-based acoustic representation, text-based visual representation, and text representation as inputs. Redundant information is eliminated through a gating mechanism to achieve effective multimodal representation interaction and fusion. The network includes a modality feature extraction layer, a gated fusion layer, and a sentiment recognition layer. The modality feature extraction layer extracts feature representations from different modalities. The gated fusion layer uses structures such as gated recurrent units (GRU) or long short-term memory networks (LSTM) to fuse features from different modalities. The sentiment recognition layer uses the fused features for sentiment classification. 3) Training and tuning: Train the model using the training set, set the loss function and optimization algorithm (Adam, SGD, etc.); optimize the model performance by adjusting hyperparameters and model structure; validate the model using the validation set, observe the model's performance on unknown data, and adjust the model parameters and structure based on the validation results; 4) Experimental setup: The dataset was divided into training, validation, and test sets to ensure that the dataset was representative and balanced; evaluation criteria were set, including accuracy, recall, and F1 score, to evaluate the model's performance; comparative experiments were conducted using TFN's tensor fusion network, LMF's low-order multimodal fusion network, and MFM's multimodal factorization model to verify the advantages of the proposed model. 5) Performance evaluation: Evaluate the model's performance on the test set, calculate and compare metrics such as accuracy, recall, and F1 score of different models; analyze the model's performance on different sentiment categories, as well as the performance differences on different modal data; 6) Results Analysis and Discussion: The experimental results are analyzed and discussed in detail, exploring the advantages and disadvantages of the model and the possible reasons; the performance of the model on different modal data is analyzed, and the impact of multimodal fusion on emotion recognition performance is discussed; comparative analysis is conducted to verify the effectiveness and advantages of the proposed model.

[0034] In this embodiment, gated recurrent networks (such as LSTM and GRU) are combined with multimodal sentiment analysis using a hierarchical fusion strategy. This fusion strategy can more effectively combine data from different modalities, thereby improving the accuracy of sentiment analysis. Through multimodal fusion, this research can provide richer and more comprehensive sentiment representations. Compared with traditional single-modal sentiment analysis, multimodal sentiment analysis can capture more dimensions of sentiment information, thus more accurately understanding and analyzing emotions. This research not only focuses on traditional sentiment analysis tasks but also explores the application of multimodal sentiment analysis in a wider range of scenarios. The expansion of these application scenarios will further promote the development and application of multimodal sentiment analysis technology.

[0035] Please see Figure 2 A schematic diagram of the second embodiment of the multimodal sentiment analysis method based on gated recurrent hierarchical fusion network provided in this invention: S11. Preprocess the text, audio, and video data respectively to obtain text semantic features, acoustic features, and visual features; S12. The DTW algorithm is used to match the timestamps of acoustic features and visual features to the timeline of text semantic features; S13 annotates the preprocessed multimodal features, constructs an annotated dataset, and divides it into training, validation, and test sets.

[0036] In this embodiment, the BERT word segmenter is used to segment the text data, convert it into word vectors, remove stop words, and use the hidden state output by the BERT word segmenter as the semantic features of the text; wavelet thresholding is used to filter noise in the audio data and extract acoustic features including Mel-frequency cepstral coefficients and fundamental frequency; OpenCV is used to extract facial key points and ResNet50 model is used to extract frame-level visual features.

[0037] In this embodiment, the original audio signal is decomposed using wavelet decomposition. A suitable wavelet basis function and decomposition level are selected to decompose the audio signal into multiple sub-band coefficients of different frequencies, including approximation coefficients and detail coefficients. By calculating the noise standard deviation of the detail coefficients at each level after wavelet decomposition, a threshold is set for each level of detail coefficients using a threshold based on Stein's unbiased risk estimation or an adaptive threshold rule to distinguish between noise and valid signal components. Coefficients with absolute values ​​less than the set threshold are set to 0, or coefficients with absolute values ​​greater than the threshold are shrunk to filter out high-frequency noise components while retaining the detail information of the valid signal. The thresholded detail coefficients are then combined with the unprocessed approximation coefficients using inverse wavelet transform to generate a denoised audio signal, completing the noise filtering process. The denoised audio signal undergoes pre-emphasis processing, using a first-order high-pass filter to enhance the high-frequency components and compensate for the attenuation of high-frequency signals during audio transmission, resulting in a flatter signal spectrum. The pre-emphasized audio signal is then framed, with each frame divided into 20-30 millisecond segments, retaining 50% overlap between adjacent frames to avoid inter-frame information loss and ensure signal continuity. A Hanning window is applied to each frame, and the Hanning window function weights the framed signal to reduce spectral leakage at frame edges, smoothing the signal within each frame and enhancing the accuracy of spectral analysis. The power of each frame is then calculated. The power spectrum is obtained by performing a Fast Fourier Transform (FFT) on the windowed frame signal to convert the time-domain signal into a frequency-domain signal, and then calculating its square value to obtain the power spectrum, which reflects the energy distribution of different frequency components. The power spectrum is then processed by a Mel filter bank, which converts the linear frequency into Mel frequencies that conform to the characteristics of human hearing by passing the power spectrum through a set of triangular Mel filters, obtaining the output energy of each filter to form the Mel spectrum. The logarithm and discrete cosine transform of the Mel spectrum are calculated, and the logarithm of the Mel spectrum is taken to convert it into a logarithmic Mel spectrum. Then, the first 12-13 coefficients are extracted by DCT to obtain the Mel cepstral coefficients, which serve as the acoustic features of the audio. The denoised audio signal is preprocessed to remove the DC component and retain low-frequency components that may contain fundamental frequency information using a low-pass filter, thus reducing interference from high-frequency noise on fundamental frequency detection. The short-time autocorrelation function (SFC) of the audio signal is calculated, and the preprocessed signal is divided into frames. The SFC for each frame is calculated; this function shows a peak at integer multiples of the fundamental frequency period, reflecting the signal's periodicity. Fundamental frequency candidate values ​​are determined by finding the time position corresponding to the first significant peak in the SFC; the reciprocal of this position is the candidate fundamental frequency value for that frame. Simultaneously, the strongest harmonic frequency is identified in the spectrum as an auxiliary candidate fundamental frequency value. Fundamental frequency tracking and smoothing are then performed. The continuity of candidate fundamental frequency values ​​between adjacent frames is checked, and outliers with excessively large jumps are removed. The fundamental frequency sequence is smoothed using a moving average filter to obtain the final fundamental frequency characteristics.

[0038] In this embodiment, a pre-trained Haar cascade classifier or a deep learning-based face detector is called in OpenCV to locate face regions from video frames. The video frames are converted into grayscale images to reduce computation and unify image channels. Simultaneously, histogram equalization is used to enhance image contrast for easier subsequent detection. The loaded face detector scans the grayscale image and outputs the bounding box coordinates of the face regions. If multiple faces exist, only the largest face region is retained. The pre-trained facial keypoint model integrated in OpenCV, such as a feature point regression model, is called. This model has been trained for feature points of key areas such as eyes, eyebrows, nose, mouth, and jaw. The detected face regions are input into the keypoint detection model, which outputs the coordinates of 68 standard facial keypoints, such as the pixel coordinates of the corners of the eyes, the tip of the nose, and the corners of the mouth, thus completing the extraction of facial keypoints. Video frames from which facial key points have been extracted are cropped into fixed-size images centered on the face, such as 224×224 pixels, ensuring the input image size meets the requirements of the ResNet50 model. The cropped images are then converted to RGB format and normalized according to the training standards of the ResNet50 model. The ResNet50 pre-trained model, which has been trained on the ImageNet dataset and contains a 50-layer convolutional neural network structure, is then used. The pre-processed image is input into the ResNet50 model, skipping the final fully connected classification layer, and the output of the feature layer after the global average pooling layer is selected as the frame-level visual feature. This feature vector contains high-level semantic information of the image. The extracted feature vector is associated with the corresponding video frame timestamp as the visual feature of that frame.

[0039] In this embodiment, the timeline of text semantic features is used as a reference benchmark. This timeline consists of timestamps of the text segmentation sequence, with each segment corresponding to the start and end timestamps in the original text. The time sequences of acoustic and visual features are extracted separately. The acoustic feature sequence is converted into a feature vector sequence corresponding to the timestamps; the visual feature sequence is also converted into a feature vector sequence. Euclidean distance is selected as the metric for measuring the similarity between acoustic feature frames and text segments, and between visual feature frames and text segments. A distance matrix is ​​constructed for acoustic and text features, and based on dynamic programming, the distance... In the matrix, a path from the starting point to the ending point is found where the elements on the path satisfy the time-increasing constraint (i.e., the timestamp of the next element is no earlier than the previous element) and the sum of all distances on the path is minimized. This path is the optimal time alignment path for acoustic or visual features and text features. Based on the optimal alignment path for acoustic features, each acoustic feature frame is matched to its corresponding text segmentation timestamp. If multiple acoustic frames correspond to the same text segmentation, they are merged into a comprehensive acoustic feature under that segmentation timestamp. Similarly, based on the optimal alignment path for visual features, each video frame is matched to its corresponding text segmentation timestamp. This achieves alignment between visual features and the text timeline. After alignment, both acoustic and visual features are based on the timestamps of text segmentation to form a multimodal feature sequence with the same length as the text sequence, ensuring that text, acoustic, and visual features correspond one-to-one in the time dimension.

[0040] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal sentiment analysis method based on gated recurrent hierarchical fusion networks, characterized in that, The method includes the following steps: S1. Multimodal data preprocessing: Collect text, audio, and video data, perform preprocessing and time-series alignment, and construct a labeled dataset; S2. Construction of Gated Recurrent Hierarchical Fusion Network: A three-level network architecture is designed, consisting of a modality feature extraction layer, a gated fusion layer, and an emotion recognition layer. The modality feature extraction layer extracts feature representations of different modal data, the gated fusion layer fuses multimodal features, and the emotion recognition layer uses the fused features to classify emotions. S3. Model Training and Optimization: The Adam optimizer is used, with the cross-entropy loss function as the target. Iterative training is performed on the labeled dataset, and overfitting is suppressed by Dropout regularization and learning rate decay. S4. Sentiment Analysis: Input the real-time collected data into the trained model, and analyze the user's emotional state in real time through the feature extraction layer, gating fusion layer and sentiment recognition layer, and output the sentiment analysis results.

2. The multimodal sentiment analysis method based on gated recurrent hierarchical fusion network as described in claim 1, characterized in that, Step S1 specifically includes the following steps: Text, audio, and video data are preprocessed separately to obtain text semantic features, acoustic features, and visual features; The DTW algorithm is used to match the timestamps of acoustic and visual features to the timeline of text semantic features; The preprocessed multimodal features are labeled to construct a labeled dataset, which is then divided into a training set, a validation set, and a test set.

3. The multimodal sentiment analysis method based on gated recurrent hierarchical fusion network as described in claim 2, characterized in that, The preprocessing of text, audio, and video data to obtain text semantic features, acoustic features, and visual features includes: The text data is segmented using the BERT word segmenter, converted into word vectors, and stop words are removed. The hidden state output by the BERT word segmenter is used as the semantic features of the text. The wavelet thresholding method is used to filter noise in the audio data and extract acoustic features including Mel-frequency cepstral coefficients and fundamental frequency. Facial key points were extracted using OpenCV, and frame-level visual features were extracted using the ResNet50 model.

4. The multimodal sentiment analysis method based on gated recurrent hierarchical fusion network as described in claim 2, characterized in that, The modality feature extraction layer includes a text feature extraction subnetwork, an audio feature extraction subnetwork, and a video feature extraction subnetwork, wherein: The text feature extraction subnetwork captures the sentiment tendency in the text based on the text's semantic features, and obtains the text feature vector; The audio feature extraction subnetwork uses 1D convolutional layers based on acoustic features to extract local temporal features in order to capture emotional cues and obtain audio feature vectors; The video feature extraction subnetwork reduces the dimensionality of visual features to 128 through pooling layers, capturing visual emotional signals including facial expressions and body movements to obtain video feature vectors.

5. The multimodal sentiment analysis method based on gated recurrent hierarchical fusion network as described in claim 4, characterized in that, The gated fusion layer includes a Bi-GRU structure and an adaptive gating mechanism.

6. The multimodal sentiment analysis method based on gated recurrent hierarchical fusion network as described in claim 5, characterized in that, The gating fusion layer employs a layered fusion strategy, which includes the following steps: During feature-level fusion, text feature vectors, audio feature vectors, and video feature vectors are concatenated and dimension-mapped to generate cross-modal feature vectors. When performing temporal hierarchical fusion, the Bi-GRU structure is used to perform temporal modeling of cross-modal feature vectors, output the temporal feature sequence of each modality, capture the temporal correlation of emotional expression, and dynamically adjust the contribution of each modality feature through gating weights; When performing decision-level fusion, global max pooling is applied to the time-series feature sequences to generate fused features.

7. The multimodal sentiment analysis method based on gated recurrent hierarchical fusion network as described in claim 1, characterized in that, The emotion recognition layer uses a fully connected layer to perform dimensionality reduction and nonlinear transformation on the fused features, outputs the emotion classification result through the softmax activation function, and predicts the emotion intensity through a regression layer. The emotion classification result includes positive, negative, and neutral.

8. The multimodal sentiment analysis method based on gated recurrent hierarchical fusion network as described in claim 1, characterized in that, Step S3 specifically includes the following steps: Set the loss function, optimizer, and hyperparameters for model training. Use the training set as input to iteratively update the model parameters. After each training round, calculate the accuracy, F1 score, and intensity prediction error on the validation set. If the F1 score on the validation set does not improve for five consecutive epochs, training is terminated and the current optimal model parameters are saved.

9. The multimodal sentiment analysis method based on gated recurrent hierarchical fusion network as described in claim 7, characterized in that, For sentiment categories that perform poorly in the validation set, the weights of the corresponding training samples are increased; the hidden dimension of the GRU in the gated fusion layer is adjusted to enhance the ability to capture temporal dependencies; and data augmentation is introduced to enhance the robustness of the model to noisy data, including at least audio noise addition and random cropping of video frames.

10. The multimodal sentiment analysis method based on gated recurrent hierarchical fusion network as described in claim 1, characterized in that, Step S4 specifically includes the following steps: Multimodal data is acquired in real time. After preprocessing the acquired real-time data, multimodal feature sequences are obtained and input into the model. The modal feature extraction layer performs in-depth extraction of text semantic features, audio acoustic features, and video visual features to enhance the emotional cues of each modality; The gated fusion layer captures the temporal dependencies of each modality feature through a bidirectional GRU and uses a genetic algorithm to optimize and obtain the modality weight coefficients. The genetic algorithm encodes each modality weight coefficient as a chromosome and randomly generates an initial population containing several weight combinations within a preset range. The selection, crossover, and mutation processes are carried out until the optimal weight combination is found. The optimal weight combination output by the genetic algorithm is used as the adjustment range constraint of the adaptive gating mechanism. Then, the weights of each modality are dynamically adjusted through the adaptive gating mechanism to generate fusion features. The emotion recognition layer processes the fused features and outputs the emotion classification result, which completes the real-time analysis of the current user's emotional state. The model's output sentiment classification results are converted into natural language descriptions, transmitted to real-world scenarios, and user evaluations of the sentiment classification results are collected. When user labeling is inaccurate, record the corresponding multimodal raw data model analysis results and the user's true sentiment to establish a user feedback database. The user feedback database is added to the labeled dataset to form an incremental training dataset. The model is then adjusted and continuously updated using the incremental training dataset.

Images