Intelligent terminal multi-mode sentiment analysis method, device and server

Abstract

The invention discloses an intelligent terminal multi-mode sentiment analysis method and device and a server, and belongs to the field of computers. The method comprises the following steps: on an intelligent terminal side, acquiring multi-modal data such as texts, voices and physiological signals and extracting features; through intra-modal adaptive attention and cross-modal bidirectional attention calculation, a fusion feature vector is generated; the emotion category and intensity are determined based on the vector, and local updates including model parameter updates and knowledge sub-graphs are generated. The local updates are sent to a federated aggregation server that constructs a global knowledge graph based on updates received from a plurality of terminals and generates a global model. The terminal receives the global model to update the local model. The invention aims to solve the problems of privacy leakage risk, inaccurate multi-modal fusion and poor model adaptability in sentiment analysis in the prior art, and achieves the technical effects of remarkably improving sentiment analysis accuracy and personalized adaptability while protecting user privacy.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a method, device, and server for multimodal emotion analysis on smart terminals. Background Technology

[0002] With the rapid development of artificial intelligence technology, human-computer interaction has become increasingly important. Among the key aspects, enabling smart devices to understand human emotions is crucial for enhancing the interactive experience. Sentiment analysis, also known as opinion mining, aims to automatically identify and analyze the emotions, viewpoints, and attitudes expressed in text, speech, or images. Accurate sentiment analysis has significant application value in scenarios such as smart terminals, mental health monitoring, and intelligent customer service.

[0003] Currently, the technologies for sentiment analysis can be mainly divided into rule-based, traditional machine learning, and deep learning methods. Deep learning-based methods, especially those using recurrent neural networks (RNNs), convolutional neural networks (CNNs), and the attention mechanism that has emerged in recent years, have shown great advantages in processing complex unstructured data.

[0004] However, the inventors discovered during their research that the aforementioned prior art solutions still have at least one of the following technical defects: High risk of privacy breaches: Traditional deep learning sentiment analysis models typically employ centralized training, requiring the uploading of raw data containing a large amount of sensitive information, such as user text and voice, to cloud servers. This process significantly increases the risk of user privacy breaches, raising serious data security and ethical concerns.

[0005] Multimodal fusion mechanisms are often crude: Human emotions are expressed collaboratively through multiple modalities (such as language, voice, facial expressions, and physiological signals). While existing technologies attempt to fuse multimodal data, the fusion methods are often simplistic, such as directly concatenating feature vectors or performing simple weighted averaging. This approach struggles to effectively capture the complex, dynamic, and nonlinear relationships between different modalities, and it cannot dynamically adjust the contribution of each modality based on data quality (such as speech in noisy environments), thus limiting the accuracy of sentiment analysis.

[0006] Poor Model Personalization and Adaptability: To address privacy concerns, federated learning has been introduced into the field of sentiment analysis. However, classic federated learning algorithms (such as FedAvg) typically employ a simple parameter averaging strategy when aggregating models from various smart terminals. This ignores the significant differences among users in their emotional expression habits, language styles, and physiological responses, resulting in a poorly adaptable global model for individual users—a "one-size-fits-all" approach that fails to achieve truly personalized sentiment understanding.

[0007] In view of this, how to design a high-precision sentiment analysis method that can protect user privacy, deeply integrate multimodal information, and adapt to individual differences among users is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0008] This application aims to provide a method, device, and server for multimodal emotion analysis on smart terminals to solve the technical problems existing in the prior art.

[0009] To achieve the above objectives, this invention proposes a multimodal sentiment analysis method for intelligent terminals. The intelligent terminal includes a local sentiment analysis model. The method includes the following steps: acquiring multimodal data from the intelligent terminal, including text, speech, and physiological signals, and extracting initial features from the multimodal data; performing intramodal adaptive attention calculation on the initial features of each modality to obtain corresponding intramodal enhanced features; performing cross-modal bidirectional attention calculation on each pair of the intramodal enhanced features to generate a fused multimodal feature vector; determining an emotion category and an emotion intensity based on the fused feature vector, and generating a local sentiment analysis model parameter update; constructing a knowledge subgraph based on the emotion category, the emotion intensity, and the fused multimodal feature vector; sending the local sentiment model parameter update and the knowledge subgraph to a federated aggregation server, so that the federated aggregation server can construct or update a global knowledge graph based on the local sentiment model parameter update and knowledge subgraph received from multiple intelligent terminal devices, and generate a global sentiment analysis model accordingly; receiving the global sentiment analysis model issued by the federated aggregation server and updating the local sentiment model.

[0010] Furthermore, the step of constructing a knowledge subgraph based on the emotion category, the emotion intensity, and the fused multimodal feature vector includes: using the emotion category or the emotion intensity as nodes, and using the fused multimodal feature vector as the attributes of the nodes; using the similarity between smart terminals as edges, and calculating the weight of the edges based on the similarity.

[0011] Furthermore, the step of sending the local sentiment model parameter update and knowledge subgraph to a federated aggregation server includes dynamically adjusting the privacy protection strategy based on the sensitivity of the data and the privacy requirements of the smart terminal.

[0012] Furthermore, the intra-modal adaptive attention calculation for the initial features of each modality includes: for the initial features of text data, processing them through an adaptive position-gated encoding and a word-speech semantic enhancement layer before performing self-attention calculation; for the initial features of speech data, processing them through an adaptive temporal encoding and a gating mechanism before performing self-attention calculation.

[0013] Furthermore, the cross-modal bidirectional attention calculation includes: for the intra-modal enhancement features of the first modality and the second modality, simultaneously calculating a first directional attention from the first modality to the second modality and a second directional attention from the second modality to the first modality; and generating cross-modal enhancement features for the first modality and the second modality based on the first directional attention and the second directional attention, respectively.

[0014] Furthermore, the method also includes: weighted fusion of the original intramodal enhancement features of a modality and multiple cross-modal enhancement features obtained by cross-modal bidirectional attention calculation with other modalities through an adaptive gating fusion module.

[0015] In another aspect, the present invention also proposes a smart terminal device, comprising: an acquisition module for acquiring multimodal data including text, voice, and physiological signals from the smart terminal; a feature extraction module for extracting initial features from the multimodal data; a feature fusion module for performing intramodal adaptive attention calculation on the initial features of each modality to obtain corresponding intramodal enhanced features; performing cross-modal bidirectional attention calculation on each pair of the intramodal enhanced features to generate a fused multimodal feature vector; and a local update generation module for determining an emotion category and a [missing information - likely a specific emotion category or function] based on the fused feature vector. The system comprises: an emotion intensity module for generating a local sentiment analysis model parameter update; a knowledge subgraph for constructing a knowledge subgraph based on the emotion category, the emotion intensity, and the fused multimodal feature vector; a federated communication module for sending the local sentiment model parameter update and knowledge subgraph to a federated aggregation server, so that the federated aggregation server can construct or update a global knowledge graph based on the local sentiment model parameter update and knowledge subgraph received from multiple smart terminal devices, and generate a global sentiment analysis model accordingly; and a model update module for updating the local sentiment model based on the global sentiment analysis model issued by the federated aggregation server.

[0016] Furthermore, when performing the cross-modal bidirectional attention calculation, the feature fusion module is specifically used to: for the intra-modal enhancement features of the first modality and the second modality, simultaneously calculate the first directional attention from the first modality to the second modality and the second directional attention from the second modality to the first modality; and generate cross-modal enhancement features for the first modality and the second modality based on the first directional attention and the second directional attention, respectively.

[0017] Furthermore, the intelligent terminal device also includes: an adaptive gating fusion module, used to perform weighted fusion of the original intramodal enhancement features of a modality and multiple cross-modal enhancement features obtained by cross-modal bidirectional attention calculation with other modalities.

[0018] In another aspect, the present invention proposes a federated aggregation server, which includes: a communication interface for receiving local sentiment analysis model parameter updates and knowledge subgraphs from multiple smart terminal devices, and sending global sentiment analysis model updates to the multiple smart terminal devices; a knowledge graph construction module for constructing or updating a global knowledge graph based on the knowledge subgraphs sent by the multiple smart terminal devices; and a global update generation module for generating the global sentiment analysis model updates based on the global knowledge graph.

[0019] Compared with the prior art, the technical solution provided by the present invention has at least the following beneficial effects: This invention achieves high-precision multimodal sentiment analysis. Addressing the issue of insufficient multimodal fusion in existing technologies, this invention first deeply mines key information within each modality through "intramodal adaptive attention," then fully captures the bidirectional interaction between different modalities through "cross-modal bidirectional attention," and finally performs intelligent weighting through "adaptive gating fusion." This hierarchical and refined fusion mechanism generates higher-quality fusion features, thereby significantly improving the accuracy of sentiment analysis.

[0020] This invention effectively protects user privacy and security. Addressing the privacy leakage risks associated with centralized training in existing technologies, this invention employs a federated learning framework. Users' raw, sensitive data, such as text, voice, and physiological signals, are always stored locally on the user's terminal and are not uploaded to the server. Only encrypted model parameter updates and abstract knowledge subgraphs are exchanged, fundamentally eliminating the risk of raw data leakage and allowing for dynamic adjustment of protection strategies based on privacy requirements.

[0021] This invention enhances the personalization and scenario adaptability of the model. Addressing the "one-size-fits-all" nature of existing federated learning models, this invention innovatively introduces a federated knowledge graph. Instead of simply averaging model parameters, the server aggregates knowledge subgraphs uploaded from various terminals to construct a global knowledge graph that reflects the commonalities and individualities of emotional expression among different user groups. The global model generated based on this knowledge graph is more targeted, enabling local models on each terminal to better adapt to users' unique emotional expression habits and dynamically changing environments, significantly improving the model's personalization level and generalization ability. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be introduced below.

[0023] Figure 1 This is an architecture diagram of a smart terminal multimodal sentiment analysis system according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a text feature adaptive attention mechanism according to an embodiment of the present invention; Figure 3 This is a flowchart of a basic attention mechanism according to an embodiment of the present invention; Figure 4 This is a schematic diagram of an adaptive attention mechanism for speech modal features according to an embodiment of the present invention; Figure 5 This is a schematic diagram of a cross-modal bidirectional fusion attention mechanism according to an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the training of a federated multimodal sentiment analysis model according to an embodiment of the present invention; Figure 7 This is a schematic diagram illustrating the training of a federated multimodal sentiment analysis model according to an embodiment of the present invention; Figure 8 This is an architecture diagram of a smart terminal according to an embodiment of the present invention; Figure 9 This is a diagram of a federated server architecture according to an embodiment of the present invention; Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in further detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0025] This invention proposes a multimodal sentiment analysis method for intelligent terminals within a federated learning framework. The following is a detailed explanation. Figure 1 Provide a detailed description of the methodology and process.

[0026] Example 1: Multimodal sentiment analysis method on smart terminal side This embodiment details the complete process of executing a multimodal sentiment analysis method on a smart terminal with a local sentiment analysis model deployed.

[0027] S101: Acquire multimodal data from the smart terminal, including text, voice, and physiological signals, and extract initial features from the multimodal data. This method first requires acquiring multi-source data necessary for sentiment analysis. Smart terminals acquire multimodal data from users through their built-in sensors (such as microphones, cameras, and heart rate sensors) and interactive interfaces (such as touchscreens). In this embodiment, the acquired multimodal data specifically includes three types: text data, voice data, and physiological signal data (such as heart rate signals).

[0028] After obtaining the raw data, it needs to be preprocessed and its features extracted to transform it into a format that can be processed by deep learning models. This process includes:

[0029] Perform operations such as word segmentation, stop word removal, and part-of-speech tagging on text data.

[0030] The speech signal is subjected to noise reduction, silence removal, and frame segmentation.

[0031] The heart rate signal is filtered to remove noise and baseline drift, and then numerically normalized.

[0032] Subsequently, corresponding deep learning models are used to extract initial features for each modality from the preprocessed data. For example, the BERT model can be used to convert text sequences into high-dimensional semantic feature vectors; a model combining CNN and Bi-LSTM can be used to extract acoustic features such as MFCC from speech frame sequences; and a one-dimensional CNN (1D-CNN) can be used to extract temporal pattern features related to emotional fluctuations from heart rate time series.

[0033] S102: Perform intra-modal adaptive attention calculation on the initial features of each modality to obtain the corresponding intra-modal enhanced features; perform cross-modal bidirectional attention calculation on each pair of intra-modal enhanced features to generate a fused multimodal feature vector; After obtaining the initial features of each modality, the core feature fusion stage begins. This stage uses a hierarchical attention mechanism to first refine the internal information of each modality, then deeply interact with the correlations between different modalities, and finally generate a high-quality fused feature.

[0034] Level 1: Intramodal Adaptive Attention Calculation This layer aims to process the initial features of each modality separately to capture its key internal information and obtain intra-modal augmented features with higher information density. This computational process is adaptable to the data characteristics of different modalities, providing high-quality input for subsequent cross-modal fusion. Specific implementations include:

[0035] Regarding the initial features of the text data: to better capture the grammatical structure and contextual dependencies in sentences, adaptive position-gated encoding and part-of-speech semantic enhancement layers are introduced for processing. The following section combines these with... Figure 2 The specific algorithm is explained below: (I) Input Embedding and Feature Enhancement 1. Text input representation: Given a sequence of tokens X

[0036] (L: length of text sequence; d) model Embedded dimensions, t1, t2...t L : Each element in the text sequence, R: represents the set of real numbers) Word embedding representation:

[0037] Among them W e b represents the word embedding weight. e This indicates the bias amount.

[0038] 2. Adaptive position-gated coding: Introducing a learnable position weight matrix Automatically adjust information weight based on word position:

[0039] in Let E represent the feature vector after incorporating adaptive position-gated encoding, and σ(⋅) be the sigmoid activation function. This indicates that the element-wise sigmoid activation function is applied to matrix P, and the "soft position" enhancement is achieved through element-wise multiplication.

[0040] 3. Part-of-speech semantic enhancement Introducing grammatical information such as nouns, verbs, and adjectives enhances the semantic representation of entity words.

[0041] in, This represents semantic enhancement features. Representing a one-hot matrix of part-of-speech tags, Represents the part-of-speech embedding matrix. Indicates the number of parts of speech categories.

[0042] (0) Improved attention mechanisms: Figure 3 This is a flowchart of the basic attention mechanism, which mainly includes the following steps: input word vectors, linear projection, Q / K / V matrix, scaled dot product attention, and output context vector. The Q, K, V mapping in the standard attention mechanism is as follows:

[0043] in, Represents a query, key, and value mapping matrix. Automatically learn important semantic dimensions based on keywords, for each token's d kAdaptive weighting of dimensions:

[0044] Represents the feature dimension weights, emphasizing important semantic dimensions; L represents the text sequence length; Q represents the feature dimension weights. i This represents the query vector corresponding to the i-th token in the sequence.

[0045] Improved calculation (weighted Q) K (Represented as follows)

[0046] Q represents the original query matrix, and K represents the original key matrix.

[0047] Semantic sparse gating (Top-k improvement) Dynamically select key connection points by combining attention scores:

[0048]

[0049] S ij A represents the attention score matrix obtained after scaling the clicks. ij This represents the sparsified attention weight matrix, retaining only the top K maximum values ​​of the attention scores.

[0050] (III) Output Generation Calculate the output vector Z, where V is the value vector:

[0051]

[0052] Finally, the final output vector H is generated through residual connections and layer normalization (LayerNorm).

[0053] in, It is the output mapping matrix.

[0054] For the initial characteristics of speech data: In order to process the continuous temporal characteristics of speech signals, adaptive timing coding and gating mechanisms are introduced.

[0055] While the attention mechanism for speech modality features and the computation of text modality features differ in their approaches to input embedding and feature enhancement, their computational methods for the improved attention mechanism are similar, and the process is as follows: Figure 4 As shown: (I): Input Embedding and Feature Enhancement Given speech signal Where T is the time step (number of frames) and D is the feature dimension (such as MFCC dimension). 1. Feature Extraction: Obtain standardized speech features using Mel-frequency cepstral coefficients (MFCC) or other feature extraction methods.

[0056] in It is the embedded dimension.

[0057] 2. Adaptive timing coding: Introducing a learnable temporal weight matrix Time-series weighting of features

[0058] Where E time Let E represent the original feature, and σ(⋅) be the sigmoid activation function. Element-wise multiplication achieves "soft temporal" enhancement.

[0059] 3. Gating mechanism combines original features with temporal augmentation features: Objective: To preserve both the integrity of the original information and the structure of the temporal enhancement features for gating decision-making.

[0060] Gating mechanisms, through adaptive selection of feature sources, preserve speech details while suppressing noise, thus significantly improving model performance. Gated signal generation: Gating weights are generated through a learnable linear transformation and a sigmoid function, enabling fine-grained control over time and feature dimensions.

[0061] in, The gating weight matrix (a learnable parameter) is used to learn the importance weight allocation rule for the two types of features; The gate bias vector (a learnable parameter) controls the threshold offset of gate activation; This indicates that the original feature E and the enhanced feature E are combined. time Concatenate into a joint feature matrix Weighted fusion: Mix the two features according to gating weights:

[0062] Where E final denoted as the final feature representation obtained by weighted mixing of the original features and temporal enhanced features, where g represents the gating weight.

[0063] If g t,d →1: The d-th dimension feature at time step t uses E time(Strengthening time-series weighting) If g t,d →0: Preserve the original feature E (avoid overcorrection) The improved attention mechanism performs some calculations similar to those used in text feature processing. The physiological signal modal feature mechanism of heart rate type fusion is as follows: 1. Location Encoding Injection: Injecting relative / absolute temporal locations into heart rate sequences lacking location information, preserving the temporal modeling capability of the features.

[0064]

[0065] This represents the encoding vector at the i-th position, where k represents the dimension index, ranging from 0 to d-1, and d is the heart rate feature dimension (such as the HRV vector dimension). This represents the original input heart rate data sequence. This represents the heart rate data sequence after adding position encoding.

[0066] I: Time step index (i-th frame), w: Sequence length (number of frames) 2. Scalar Self-Attention Calculation

[0067] Let d represent the learnable weight matrices for the query, key, and value, respectively. k For attention head dimension (general) ), This is the temporal context representation after weighted fusion.

[0068] 3. Enhanced Key Patterns Based on self-attention, the key pattern score of each frame is weighted twice:

[0069]

[0070] This represents the enhanced heart rate feature vector, β. i Let s be the attention weight for the i-th feature. i Let be the unnormalized attention score of the i-th feature. MLP stands for Multilayer Perceptron, used to compute the attention score.

[0071] Frame-level importance scores are calculated using MLP, where βi represents the frame-level attention weight, highlighting the heart rate segments most sensitive to emotion.

[0072] The self-attention mechanism of the heart rate modality can correctly extract the internal temporal relationships and generate sentiment feature vectors that can be used for cross-modal fusion in federated learning.

[0073] Level 2: Cross-modal bidirectional attention computation After obtaining the intramodal enhancement features of each modality, cross-modal bidirectional attention calculation will be performed on these features to fully explore the complementary and correlated information between different modalities.

[0074] The computation is performed through a "cross-modal bidirectional attention block," which simultaneously performs attention computation in two directions for any pair of modalities (e.g., modality A and modality B): attention from modality A to modality B and attention from modality B to modality A. This bidirectional information flow design ensures that the interaction between modalities is equal and reciprocal, thereby avoiding the information bias that may be caused by unidirectional fusion.

[0075] like Figure 5 As shown, since this embodiment involves three modalities—text, speech, and heart rate—the bidirectional attention computation is applied to all possible modal pairs: text-speech, text-heart rate, and speech-heart rate. During this process, each modality interacts with the other two modalities, thus obtaining two independent augmented features. For example, the text modality obtains one augmented feature from its interaction with speech and another from its interaction with heart rate. Similarly, the speech and heart rate modalities each obtain two augmented features derived from the other modalities.

[0076] Finally, to form the final representation for each modality, the original features of that modality (i.e., the features computed after intramodal self-attention) need to be fused with two enhanced features obtained from cross-modal interactions. This fusion can be accomplished through a gating mechanism or a weighted summation. Finally, the final representations of the three modalities are concatenated to form a unified feature vector, which is then input into the sentiment analysis classifier.

[0077] This method also includes a follow-up step: an adaptive gating fusion module performs a final weighted fusion of the original intra-modal augmentation features of a modality, as well as multiple cross-modal augmentation features obtained by cross-modal bidirectional attention computation with all other modalities. This module can dynamically learn to determine the reliability and importance of augmentation information from different sources and assign corresponding weights, thereby generating the most robust and comprehensive final representation of the modality.

[0078] The specific algorithm for the above steps is as follows: - Local sentiment classifier: Outputs sentiment categories (such as happy, angry, sad, etc.) or sentiment dimensions (VAD).

[0079] The specific process is as follows: 1. Input Feature Representation: Based on the three modal features obtained from the intramodal self-attention calculation above. Text features: (L) t : Length of the text sequence, d t (Feature dimension) Speech features: (L) a : Length of the speech sequence, d a (Feature dimension) Heart rate characteristics: (L) h Heart rate feature dimension, d h (Feature dimension) 2. Cross-modal bidirectional attention mechanism It enables bidirectional information exchange between modalities, solving the information bias problem of traditional unidirectional attention.

[0080] Taking text-to-speech modality as an example: Step 1: Feature Alignment Projection Different modalities can be projected onto a unified space using learnable matrices:

[0081] Where T' and A' represent the matrices of the text modality and the speech modality after feature alignment and projection, respectively, and W t and W a This represents a learnable weight matrix used to project features from different modalities into a unified space, d t and d a These represent the original feature dimensions for text and audio modalities, respectively.

[0082] Step 2: Bidirectional Attention Calculation Text to Speech (T→A):

[0083]

[0084] Among them, V a This means that the text-to-speech attention output is obtained by multiplying the speech features as the value vector and the attention weights by the value vector.

[0085] Voice to Text (A→T):

[0086]

[0087] Among them, V tThis means that the text features are used as the value vector, and the attention output from speech to text is obtained by multiplying the attention weights with the value vector.

[0088] Step 3: Residual Fusion and Gating

[0089] Among them G t G a It is a learnable gating vector that controls the intensity of information fusion.

[0090] Each modality acquires two augmentation features (e.g., text augmentation features). (From voice) (From heart rate) 3. Adaptive Gating Fusion Module Dynamic weighted fusion of multi-source enhancement features of the same modality solves the problems of redundancy and conflict.

[0091] Step 1: Process the original features T′ and enhanced features obtained in the previous process. , Global Average Pooling (GAP) is used to obtain the feature vectors t' and t'. a ,t h Step 2: Gating Weight Generation

[0092]

[0093] Among them W g W z g is a learnable parameter self G represents the original feature gating weights. a Indicates enhanced feature t a gating weight, g h Indicates enhanced feature t h gating weight, b g This represents the bias term of the gating layer, used to adjust the output of the gating mechanism, b z This represents the bias term of the nonlinear transformation layer, used for generating the gate weights.

[0094] Step 3: Weighted fusion to obtain text fusion features:

[0095] Repeat the same operation on the speech and heart rate modalities to obtain speech fusion feature A. fused Heart rate fusion feature H fused 4. Multimodal feature fusion Step 1: Fuse features T for each modality fused A fused H fused Global Average Pooling (GAP) is used to obtain the feature vectors t, a, h.

[0096] Step 2: Cross-modal collaborative fusion

[0097] Where ⊕ represents a splicing operation. For element-wise multiplication, λ is a learnable scalar.

[0098] Note: The credibility score calculation network learns during training and can dynamically adjust weights based on the quality of the input data (e.g., when the speech signal is contaminated with noise, the weight of the speech modality can be reduced).

[0099] After completing the above operations, local sentiment prediction can be performed, such as using a fully connected layer to output sentiment category and intensity:

[0100] F is the feature vector after cross-modal feature fusion.

[0101] The loss function can be calculated using cross-entropy plus a sentiment consistency regularization term:

[0102] in, This is the prediction result from the previous round of global model. Cross-entropy loss function measures the difference between the probability distribution predicted by the model and the true label.

[0103] After all modalities have been fused as described above, their final representations are concatenated or passed through a fully connected layer again to generate a unified, highly condensed fused multimodal feature vector.

[0104] S103: Based on the fused feature vector, determine an emotion category and an emotion intensity, and generate a local sentiment analysis model parameter update; construct a knowledge subgraph based on the emotion category, the emotion intensity, and the fused multimodal feature vector; Based on the fused multimodal feature vector generated in the previous step, the local sentiment analysis model will perform the following operations: Sentiment determination: The fused feature vector is input into the prediction layer of the model to determine a sentiment category (discrete value, such as happy or sad) and a sentiment intensity (continuous value, such as a rating between 0 and 1).

[0105] Model parameter update generation: By comparing the predicted results with (if any) the true labels, the loss function is calculated, and the gradient of all trainable parameters of the local model is calculated using the backpropagation algorithm. This gradient is the local sentiment analysis model parameter update.

[0106] Knowledge Subgraph Construction: Simultaneously, the model constructs a structured knowledge subgraph based on the key information from this analysis. This construction process specifically includes: using the sentiment category or sentiment intensity obtained from this analysis as a node in the graph, and using the fused multimodal feature vector that generated the result as an attribute of that node. Furthermore, the graph can also contain edges, which can represent the similarity between smart terminals (e.g., by comparing the feature vectors or model parameters of different terminals), and the edge weights are calculated based on the similarity.

[0107] S104: The local sentiment model parameter update and knowledge subgraph are sent to a federated aggregation server, so that the federated aggregation server can construct or update a global knowledge graph based on the local sentiment model parameter update and knowledge subgraph received from multiple smart terminal devices, and generate a global sentiment analysis model accordingly. The smart terminal sends two parts: local sentiment model parameter updates and a knowledge subgraph, to the federated aggregation server. During transmission, the system can dynamically adjust its privacy protection strategy based on the sensitivity of the data (e.g., the analysis results involve highly private sentiment) and the privacy requirements of the smart terminal. For example, it can use stronger differential privacy noise for more sensitive data, or use more complex encryption protocols based on cryptographic hashing, secure multi-party computation, etc., to achieve flexible and adaptive privacy protection. The federated aggregation server constructs or updates a global knowledge graph based on the local sentiment model parameter updates and knowledge subgraphs received from multiple smart terminal devices, and generates a global sentiment analysis model accordingly.

[0108] S105: Receive the global sentiment analysis model issued by the federated aggregation server and update the local sentiment model.

[0109] In the next round of federated learning, the smart terminal will receive a new generation of globally optimized global sentiment analysis model from the federated aggregation server. After receiving the model, the smart terminal will use it to update its own local sentiment analysis model, for example, by setting the weights of the local model to the weights of the global model, thus completing a learning loop.

[0110] Example 2: Smart Terminal Device This embodiment provides a smart terminal device 800, which is a physical carrier for performing the method described in Embodiment 1. The smart terminal device structurally includes the following functional modules, such as... Figure 8 As shown:

[0111] The acquisition module 801 is used to perform the data acquisition steps in the aforementioned method, specifically to acquire multimodal data from the smart terminal, including text, voice and physiological signals.

[0112] A feature extraction module 802 is used to perform the feature extraction steps in the aforementioned method, specifically to extract the initial features of the multimodal data.

[0113] A feature fusion module 803 performs the hierarchical feature fusion step described in the aforementioned method. Specifically, this module performs intra-modal adaptive attention calculation on the initial features of each modality to obtain intra-modal enhanced features, and then performs cross-modal bidirectional attention calculation on each pair of these intra-modal enhanced features to ultimately generate a fused multimodal feature vector. When performing cross-modal bidirectional attention calculation, this module simultaneously calculates the first directional attention from the first modality to the second modality and the second directional attention from the second modality to the first modality, and generates cross-modal enhanced features for each modality based on these two directional attentions.

[0114] The adaptive gating fusion module 804 can be regarded as part of the feature fusion module or an independent subsequent module. Its function is to perform weighted fusion of the original intramodal augmentation features of a modality and multiple cross-modal augmentation features obtained by interacting with other modalities.

[0115] A local update generation module 805 is used to execute the local update generation steps in the aforementioned method. Specifically, it is used to determine the sentiment category and sentiment intensity based on the fused feature vector, generate local sentiment analysis model parameter updates, and construct a knowledge subgraph.

[0116] A federated communication module 806 is responsible for communicating with the federated aggregation server. Specifically, it is used to send local sentiment model parameter updates and knowledge subgraphs to the server, and to receive the global sentiment analysis model issued by the server.

[0117] A model update module 807 is used to execute the local model update step in the aforementioned method, specifically to update the local sentiment analysis model based on the received global sentiment analysis model.

[0118] Example 3: Federated Aggregator Server This embodiment also provides a federated aggregation server 900, such as Figure 9 As shown, this server serves as the coordination hub for the entire federated learning system. The federated aggregation server structurally includes the following functional modules:

[0119] A communication interface 901 functions as a portal for data exchange between the server and multiple smart terminal devices. Specifically, it receives local sentiment analysis model parameter updates and knowledge subgraphs from multiple smart terminal devices, and after completing aggregation optimization, sends newly generated global sentiment analysis model updates to these devices.

[0120] A knowledge graph construction module 902 is designed to build or update a global knowledge graph based on knowledge subgraphs sent from all participating smart terminal devices. This module can extract universal, cross-user "emotional knowledge" and discover personalized emotional expression patterns among different user groups by aligning, fusing, and reasoning through massive amounts of local knowledge subgraphs (e.g., using graph neural networks, GNNs).

[0121] A global update generation module 903 is responsible for generating the final global sentiment analysis model update. While performing routine model parameter aggregation (such as FedAvg), this module leverages insights from the global knowledge graph generated by the knowledge graph construction module to guide and optimize the aggregation process. The resulting global model not only integrates model parameters from all users but also incorporates "wisdom" extracted from global knowledge, thus exhibiting stronger performance and adaptability.

[0122] This invention also provides a federated aggregation server as the coordination hub of the entire federated learning system. Structurally, this server includes a communication interface, a knowledge graph construction module, and a global update generation module, which work together to achieve intelligent aggregation of information uploaded from various terminals.

[0123] The server's workflow begins with receiving local sentiment analysis model parameter updates and knowledge subgraphs from multiple smart terminals via its communication interface. All uploaded information has undergone appropriate privacy protection processing on the smart terminal side.

[0124] Next, the server enters its core process of federated knowledge fusion and global dynamic graph construction and update. The server uses federated learning algorithms (such as FedAvg) to aggregate the information uploaded by each smart terminal. This aggregation process is not simply averaging model parameters; its more important aspect lies in the structured fusion of the knowledge subgraphs uploaded by each party. During this process, the server performs a weighted average of the feature vectors of common nodes and the weights of edges, thereby constructing a preliminary global knowledge graph. The entire process strictly adheres to the principle of adaptive privacy protection mechanisms. The information processed by the server has already been preprocessed by the smart terminals according to the data's sensitivity and privacy requirements, using techniques such as encrypted hashing, differential privacy, or secure multi-party computation, thus ensuring that users' original privacy information is effectively protected while knowledge is shared.

[0125] After obtaining the initial global knowledge graph, the server performs global dynamic knowledge graph optimization. For example, the server can internally use a graph neural network (GNN) to further learn and reason about the aggregated global knowledge graph to uncover deeper, cross-user representations of nodes and edges, thereby capturing more complex group sentiment patterns. This optimized global knowledge graph will serve as the foundation for generating the global model in this round of federated learning and will provide higher-quality prior knowledge for the next round of learning.

[0126] Finally, the server's global update generation module generates the final global sentiment analysis model update based on the aggregated model parameters and the optimized global knowledge graph. This update is distributed to each smart terminal via the communication interface through a model update and feedback mechanism. The smart terminal updates its local model according to this global model, thus completing one iteration. Through this iterative process, the sentiment analysis performance and generalization ability of the entire system are gradually improved.

[0127] This embodiment further elaborates on the collaborative working method of the system from the perspective of the overall training process, such as... Figure 6 As shown, the entire training process is distributed, and its core objective is to leverage the server-side fused global knowledge graph (or its embedded representation) to enhance the local multimodal sentiment analysis model of each participant (i.e., the smart terminal).

[0128] To achieve this goal, each smart terminal's local model architecture includes a knowledge injection module in addition to a multimodal feature encoder for processing raw inputs or features such as text, speech, and physiological signals. This module integrates federated global knowledge graph knowledge obtained from the server into the local model's analysis and decision-making process. Its implementation methods vary, with common ones including knowledge-aware attention, which uses information from the global knowledge graph to dynamically adjust the local model's attention weights for different modalities or features within a modality, enabling the model to focus more on key information consistent with global knowledge. Another approach is knowledge graph convolution, which treats locally extracted multimodal features as temporary nodes in a graph and combines them with relevant entities or relationships retrieved from the global knowledge graph to form a hybrid graph. Then, a graph neural network (GNN) model is used for message passing and aggregation on this graph, effectively integrating global knowledge into the local feature representation. Alternatively, a direct approach of knowledge feature concatenation or fusion can be used, concatenating the embedding vectors of the retrieved entities or relationships with the original local modal features, or fusing them through a learnable gating mechanism.

[0129] After acquiring knowledge injection capabilities, the federated training process iterates through the following steps. At the start of each training round, each participant uses its local multimodal data and the latest federated global knowledge graph representation obtained from the server to train or fine-tune its sentiment analysis model locally. After local training is complete, each participant only uploads parameter updates (such as gradients or weight changes) to the coordination server; the user's original data and local knowledge graph itself never leave the device. The server securely aggregates all model updates uploaded by all participants, generating a new, more powerful global sentiment analysis model, and distributes it to all participants for use in the next training round. The system repeats this cycle until the performance of the global model converges and stabilizes. The key point of this process is that model training itself is completely distributed, achieving no sharing of raw data and local knowledge graphs; only encrypted updates of model parameters are shared and aggregated within the federated network, thus ensuring a high degree of privacy and security.

[0130] To further enhance the intelligence of the model, the present invention may also include a cross-modal knowledge transfer mechanism, the purpose of which is to share deep correlation knowledge between specific modalities discovered in some smart terminals with all smart terminals through a federated network.

[0131] The mechanism first extracts cross-modal knowledge locally on the smart terminal. In the local model of each smart terminal, the system extracts emotional feature vectors from different modalities (such as text and speech) and calculates the similarity or correlation between them. Through this calculation, some emotional association patterns unique to the user can be identified. For example, when a user uses a certain sarcastic tone, the wording of their text also shows a certain pattern.

[0132] When performing cross-modal knowledge fusion on the server side, the federated aggregation process not only aggregates model parameters but also merges cross-modal relationship patterns uploaded from different smart terminals, thereby forming a richer knowledge base about cross-modal relationships in the global knowledge graph. Through this global cross-modal knowledge graph, the sharing of emotional knowledge between different modalities can be realized, improving the system's adaptability to complex emotional expressions.

[0133] Finally, knowledge application is achieved through dynamic knowledge transfer. After each federated update is issued by the server, the smart terminal can dynamically adjust the intermodal attention weights of its local model based on the latest cross-modal knowledge obtained from the global knowledge graph.

[0134] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal sentiment analysis method for a smart terminal, wherein the smart terminal includes a local sentiment analysis model, characterized in that, Includes the following steps: Acquire multimodal data from the smart terminal, including text, voice, and physiological signals, and extract initial features from the multimodal data; For each modality, intra-modal adaptive attention is calculated on the initial features to obtain the corresponding intra-modal enhanced features; Cross-modal bidirectional attention is calculated between each pair of the intramodal enhancement features to generate a fused multimodal feature vector; Based on the fused feature vector, an emotion category and an emotion intensity are determined, and a local sentiment analysis model parameter update is generated; a knowledge subgraph is constructed based on the emotion category, the emotion intensity, and the fused multimodal feature vector. The local sentiment model parameter updates and knowledge subgraphs are sent to a federated aggregation server, which then constructs or updates a global knowledge graph based on the local sentiment model parameter updates and knowledge subgraphs received from multiple smart terminal devices, and generates a global sentiment analysis model accordingly. Receive the global sentiment analysis model issued by the federated aggregation server and update the local sentiment model.

2. The method according to claim 1, characterized in that, The construction of a knowledge subgraph based on the emotion category, the emotion intensity, and the fused multimodal feature vector includes: The emotion category or emotion intensity is used as a node, and the fused multimodal feature vector is used as the attribute of the node; the similarity between smart terminals is used as an edge, and the weight of the edge is calculated based on the similarity.

3. The method according to claim 1, characterized in that, The step of sending the updated local sentiment model parameters and knowledge subgraph to a federated aggregation server includes dynamically adjusting the privacy protection strategy based on the sensitivity of the data and the privacy requirements of the smart terminal.

4. The method according to claim 1, characterized in that, The process of performing intra-modal adaptive attention calculation on the initial features of each modality includes: Based on the initial features of the text data, an adaptive position-gated encoding and a one-word semantic enhancement layer are used for processing before self-attention calculation is performed; The initial features of the speech data are processed through an adaptive temporal coding and a gating mechanism before self-attention calculation is performed.

5. The method according to claim 1, characterized in that, The cross-modal bidirectional attention calculation includes: For the intra-modal enhancement features of the first mode and the second mode, a first directional attention from the first mode to the second mode and a second directional attention from the second mode to the first mode are calculated simultaneously. Based on the first directional attention and the second directional attention, cross-modal enhancement features are generated for the first modality and the second modality, respectively.

6. The method according to claim 5, characterized in that, The method further includes: An adaptive gating fusion module is used to weight and fuse the original intramodal enhancement features of a modality and multiple cross-modal enhancement features obtained by cross-modal bidirectional attention calculation with other modalities.

7. A smart terminal device, characterized in that, include: An acquisition module is used to acquire multimodal data from the smart terminal, including text, voice, and physiological signals. A feature extraction module is used to extract initial features from the multimodal data; A feature fusion module is used to perform intra-modal adaptive attention calculation on the initial features of each modality to obtain the corresponding intra-modal enhanced features; Cross-modal bidirectional attention is calculated between each pair of the intramodal enhancement features to generate a fused multimodal feature vector; A local update generation module is used to determine an emotion category and an emotion intensity based on the fused feature vector, and generate a local sentiment analysis model parameter update; and to construct a knowledge subgraph based on the emotion category, the emotion intensity, and the fused multimodal feature vector. A federated communication module sends the local sentiment model parameter updates and knowledge subgraph to a federated aggregation server, so that the federated aggregation server can construct or update a global knowledge graph based on the local sentiment model parameter updates and knowledge subgraph received from multiple smart terminal devices, and generate a global sentiment analysis model accordingly. as well as A model update module is used to update the local sentiment model based on the global sentiment analysis model issued by the federated aggregation server.

8. The intelligent terminal device according to claim 7, characterized in that, The cross-modal bidirectional attention calculation includes: For the intra-modal enhancement features of the first mode and the second mode, a first directional attention from the first mode to the second mode and a second directional attention from the second mode to the first mode are calculated simultaneously. Based on the first directional attention and the second directional attention, cross-modal enhancement features are generated for the first modality and the second modality, respectively.

9. The method according to claim 7, characterized in that, The method further includes: The adaptive gating fusion module is used to perform weighted fusion of the original intramodal enhancement features of a modality and multiple cross-modal enhancement features obtained by cross-modal bidirectional attention calculation with other modalities.

10. A federated aggregation server, characterized in that, include: A communication interface is used to receive local sentiment analysis model parameter updates and knowledge subgraphs from multiple smart terminal devices, and to send global sentiment analysis model updates to the multiple smart terminal devices. A knowledge graph construction module is used to construct or update a global knowledge graph based on the knowledge subgraphs sent by the multiple smart terminal devices; and A global update generation module is used to generate updates to the global sentiment analysis model based on the global knowledge graph.

Images