Graph structure attention multi-mode emotion recognition method fusing social attributes

Abstract

The invention discloses a multi-modal intelligent recognition method for emotion calculation, which comprises the following steps: acquiring multi-source emotion signals such as voice, text and facial expression, respectively extracting visual features such as acoustic features, semantic vectors and action units, and further constructing a weighted graph structure associated with fused modal similarity and social attributes; according to the system, local neighborhood feature propagation is achieved through a graph neural network, the incidence relation between samples in the aspects of emotional expression, expression habits and social background is captured, global emotional dependence is coded in combination with a Transform model, and deep emotional representation with cross-modal consistency and individual difference adaptability is obtained. And finally, realizing emotion category, emotion dimension or psychological state recognition through an emotion prediction module. The method can keep stable performance under different user groups and complex interaction scenes, and has high generalization and application value.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and affective computing, focusing on a graph structure deep learning modeling method for multimodal emotional data, and providing a graph structure attention-based multimodal emotion recognition method that combines social attribute information. By constructing a heterogeneous data graph structure containing speech, text, and visual features, and utilizing the joint attention mechanism of graph neural networks and Transformers, efficient modeling and automatic recognition of complex emotional states are achieved. Background Technology

[0002] Affective computing aims to enable computers to recognize, understand, and analyze human emotions, and is an important interdisciplinary research direction in artificial intelligence, cognitive science, and psychology. With the rapid development of deep learning technology, sentiment analysis has gradually expanded from traditional tasks such as single-modal speech recognition, text sentiment classification, and facial expression recognition to multimodal emotion recognition that comprehensively analyzes multimodal cues such as speech intonation, language semantics, and visual expressions.

[0003] Multimodal methods can capture complementary information between different modalities and are considered an important way to improve the performance of emotion understanding tasks.

[0004] Existing multimodal emotion recognition methods typically extract features from each modality based on CNN, RNN, or Transformer models, and achieve information fusion through feature concatenation, self-attention mechanisms, or cross-modal alignment.

[0005] However, emotional expression itself exhibits significant individual variability and social attribute dependence, with distinct differences among different groups in their emotional expression methods, language styles, and speech features. Furthermore, complex semantic dependencies exist between multimodal data; for example, text semantics may not align with speech prosody, and facial movements and verbal expressions may be out of sync. This makes it difficult for existing models relying on simple fusion mechanisms to adequately model deep cross-modal relationships.

[0006] To further enhance modeling capabilities, some studies have introduced Graph Neural Networks (GNNs), which utilize graph structures to express the relationships between samples, modalities, or frames.

[0007] However, current GNN-based methods mainly focus on internal modal relationships or sequence structure relationships, and rarely consider "social attributes" as key factors of graph structure nodes. Social attributes (such as gender, age, educational background, cultural differences, etc.) have a significant influence on emotional expression style and the way depressive tendencies are presented.

[0008] GNNs still have limitations in handling long-distance cross-modal dependencies, and their local aggregation characteristics make it difficult to capture global semantic relevance.

[0009] Meanwhile, while Transformer possesses powerful global modeling and cross-modal alignment capabilities, its input structure typically relies on tiling sequences, lacking the ability to explicitly express the topological structure between modalities and the social relationships between samples, making it difficult to effectively utilize the naturally existing graph structure characteristics in multimodal data. Due to these limitations, existing technologies often suffer from performance degradation and insufficient generalization ability when facing complex scenarios (such as emotion recognition or psychological state assessment across groups and contexts).

[0010] In summary, the current field of affective computing still lacks a unified deep learning framework that can effectively integrate cross-modal semantic dependencies of multimodal data, and introduce social attributes to model individual differences and combine graph structure representation with global attention mechanisms.

[0011] This invention proposes a novel, multimodal fusion emotion recognition algorithm to improve the model's ability to express and generalize multimodal emotion data in complex social contexts. Summary of the Invention

[0012] This invention proposes a Graph-Transformer multimodal emotion recognition method that incorporates social attribute information. The overall approach is as follows: First, modality-specific encoding is performed on multi-source data such as speech, text, and visual facial expressions. Then, the encoding results of each modality and the social attribute information of the subjects are mapped together as graph nodes and attribute nodes. A weighted adjacency matrix is ​​constructed through a similarity calculation based on nonlinear kernels and attribute similarity constraints. Subsequently, a graph neural network is used on this graph structure to propagate local structured features. Finally, the node representations output by the GNN are input into the Transformer to capture global semantic dependencies across modalities and predict emotions or depressive tendencies.

[0013] This method explicitly introduces social attributes at the structural level, enabling the model to capture the differences in emotional expression among different individuals or groups while preserving semantic information between modalities, thereby improving recognition accuracy and generalization ability.

[0014] In terms of data processing and feature extraction, this invention cleans and encodes each modality according to a preprocessing procedure: the speech stream generates acoustic vector representations through self-supervised acoustic models such as wav2vec; the text data obtains semantic embedding vectors through BERT-like pre-trained language models; and facial expressions extract action units and temporal features through OpenFace.

[0015] For each sample i, denote the set of modal feature vectors obtained after encoding and form a unified node feature representation h.i The node features of all samples are stacked row-wise to form a feature matrix H = [h1; h2; ...; h n This provides input for subsequent graph structure modeling.

[0016] To characterize the similarity relationships between samples, this invention first calculates the Euclidean distance matrix of node features, denoted as D, whose elements are the Euclidean distances between two node features: d ij =||h i -h j ||2.

[0017] Based on the distance matrix, a radial basis function (RBF) kernel is introduced for nonlinear mapping to obtain the feature similarity matrix S, whose elements are defined as follows:

[0018] The kernel width parameter σ can be selected as a statistic of the feature matrix H to achieve a balance between local and global relationships, thereby improving the ability to characterize similarity in high-dimensional space.

[0019] Building upon this, to integrate social attribute information into the graph topology, this invention designs a method for constructing the attribute similarity matrix. Taking gender and age as an example, the gender similarity matrix G can be defined as follows:

[0020] and age similarity matrix The parameter τ>0 controls the rate at which age differences diminish the similarity.

[0021] For cases with more social attributes, the similarity of each attribute can be combined in a weighted manner to obtain the overall attribute similarity matrix S. (attr) : Where w is the normalized weight of each attribute, which can be set prior or learned through training.

[0022] Subsequently, the present invention compares feature similarity S with attribute similarity S. (attr) The final weighted adjacency matrix A is formed by fusing the features / attributes according to the attention coefficients. Let λ∈[0,1] be the feature / attribute fusion coefficients, then A=λS+(1-λ)S (attr) λ can be a hyperparameter or a learnable attention weight so that the model can adaptively adjust the relative importance of feature similarity and social attribute similarity in different datasets or scenarios.

[0023] To ensure numerical stability and controllability of graph convolution, the adjacency matrix can be further normalized symmetrically: Where D is the degree matrix of A, D ii =∑ j A ij The combined construction of feature similarity and attribute similarity allows the graph topology to reflect both the semantic proximity of modal expressions and the potential connections between individuals guided by social attributes.

[0024] In the graph neural network module, this invention is based on a normalized adjacency matrix. Node features are propagated and aggregated. Taking a two-layer GCN as an example, let the initial node features be H. (0) =H, the update rule for the l-th layer is Among them W (l) Let H be the linear mapping matrix of the l-th layer, and σ(·) be the nonlinear activation function (such as ReLU). The H obtained after two iterations... (2) It can fuse neighborhood information with node information to generate high-order node representations containing structured context, providing high-quality input for subsequent global modeling. Attention weighting (such as GAT) can be used in the GCN layer to make the edge weights learnable, thereby further improving the adaptive ability of attribute and feature fusion.

[0025] Considering that GCN excels at local structure modeling while Transformer excels at capturing global dependencies, this invention sets up a Transformer encoder after GCN to perform global semantic interactions on graph node representations. To balance graph structure information and self-attention mechanisms, this invention can use a graph-aware Transformer: adjacency information is added as an attention bias term when calculating self-attention weights, so as to preserve the constraints of graph topology while maintaining the ability to capture global information. Formally, the attention score for a pair of nodes can be expressed as:

[0026] Where is a linear mapping of Q and K, and d k η is the scaling factor, and η is the adjacency bias weight coefficient. The Transformer layer further integrates long-distance semantic relationships between nodes through multi-head attention and a feedforward network, and finally outputs a global discriminant vector after pooling or node-level projection for downstream classification or regression tasks.

[0027] To improve the interpretability and social consistency of the model, this invention incorporates an attribute similarity regularization term into the training objective, encouraging the maintenance of representational similarity among samples with similar attributes. Let the classification (or regression) loss be... (e.g., cross-entropy or mean squared error), the attribute similarity regularization term is defined as Where z i This represents the low-dimensional discriminative representation of the model's final output. The overall training loss is... Where γ is the regularization weight, λ reg Let θ be the coefficient of the regularization term, and θ be the set of model parameters. Through this joint optimization, the model improves prediction accuracy while also taking into account the consistency constraints of social attributes, reducing bias and enhancing cross-population generalization.

[0028] In terms of implementation, this invention also stipulates several engineering details for reproducibility: in the feature extraction stage, a pre-trained model can be used for transfer learning to alleviate the scarcity of annotations; in the graph construction stage, a sparsity strategy can be introduced to control computational complexity; training adopts a mini-batch approach combined with graph sampling to handle large-scale datasets; the optimizer adopts AdamW, and hyperparameters such as learning rate, γ, and λ are determined through cross-validation or Bayesian search. Attached Figure Description

[0029] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0030] This implementation is based on a multimodal emotion computing scenario. By constructing a multi-source emotion cue processing pipeline including voice, text, and facial expressions, it achieves automatic recognition and representation of complex human emotions. The system architecture of this invention consists of a multimodal feature extraction module, a social attribute processing module, a graph structure construction module, a local graph neural network feature propagation module, a global Transformer representation module, and an output inference module at the software level. At the hardware level, it is deployed in a conventional GPU server environment, but can also be ported to edge computing devices, personal computers, or mobile platforms, without limiting the specific hardware configuration.

[0031] In actual operation, the system receives raw input from different modalities according to a unified data processing flow. Speech data can come from microphones, multi-channel recording devices, or speech tracks in audio and video files; text data can come from dialogue records, user input, or speech-to-text results; facial expression data comes from continuous video frames captured by a camera. The system uses an internal scheduling module to ensure that the data from each modality are aligned in the time dimension and to provide a structured input foundation for subsequent multimodal feature fusion.

[0032] In the speech processing stage, this implementation first performs noise reduction, sampling rate conversion, and amplitude normalization on the original speech to ensure a good signal-to-noise ratio for the input signal. Then, short-time energy detection is used to remove silent segments, avoiding redundant segments from causing training bias in the model. The preprocessed speech segments are re-divided into fixed-length windows, and each window is input into a pre-trained wav2vec model to obtain a high-dimensional latent representation of the speech. The system extracts the hidden vectors from the middle or last layer of the wav2vec model as acoustic emotion features, including prosodic rhythm, speech rate variations, pitch fluctuations, and energy intensity, among other potential emotional cues. To ensure that speech inputs of different lengths maintain consistent dimensionality in subsequent modules, the system remaps variable-length features to fixed-length vectors using average pooling or interpolation strategies.

[0033] The text modality processing employs a natural language processing pipeline, including regularization cleaning, punctuation preservation, stop word filtering, and word segmentation based on sub-word units. The processed text sequence is mapped to token IDs acceptable to the BERT model, and context-dependent semantic representations are obtained by calling BERT's Transformer encoder. This implementation uses the [CLS] bit output as the global representation of text sentiment features, while retaining the token representations of intermediate layers for subsequent expansion, such as the analysis of fine-grained sentiment components. In the case of multi-sentence or long text inputs, the system automatically segments the text, extracts semantic features at the sentence level, and then generates a uniform-length text sentiment vector through sentence vector averaging or attention weighting.

[0034] Facial expression modality data is derived from video frame sequences. During the input phase, the system first uses a face detection module to detect and align each frame to reduce the impact of pose differences. Subsequently, the OpenFace module extracts visual emotion features such as action units, facial muscle activation levels, and geometric keypoint positions. In this implementation, after extracting AU features at the frame level, a sliding time window strategy is used to smooth consecutive frames to suppress noise and feature jumps caused by occasional expression changes. Finally, the system maps variable-length video segments into fixed-length visual emotion vectors using methods such as temporal average pooling, BiLSTM aggregation, or self-attention compression to adapt to the graph structure construction and feature fusion steps.

[0035] After extracting independent features from the three modalities, the system concatenates or projects the speech features, text features, and facial expression features according to their feature dimensions, forming a unified-dimensional multimodal node feature vector. To eliminate the problem of inconsistent feature dimensions across different modalities, this implementation performs feature normalization and standardization on each modal component, ensuring comparability of all features. The unified node features form a matrix H = [h1, h2, ..., h...]. n], which serves as the basic input for subsequent graph structure generation.

[0036] In addition to modal features, this implementation also supports the structured input of users' basic social attributes, including gender, age, educational background, and cultural region. Social attribute fields are mapped by the system to quantifiable features for subsequent construction of a social attribute similarity matrix. Discrete attributes such as gender are processed using one-hot encoding, while continuous attributes such as age are normalized and numerically mapped using a distance function. All social attribute features undergo structured transformation during the preprocessing stage and are used as additional node features or to supplement similarity modeling between nodes.

[0037] Through the above steps, this implementation achieves comprehensive preprocessing and feature unification of multimodal input signals, enabling the system to fuse and model multimodal emotional information from a graph structure perspective in subsequent steps. This implementation does not limit the number of input modalities and is compatible with situations where acoustic modalities, text modalities, or visual modalities are missing. A dynamic masking mechanism ensures the system functions correctly in different scenarios.

[0038] After preprocessing multimodal features and social attributes, this implementation proceeds to the graph structure construction stage. This stage aims to map previously independent samples or data fragments into unified graph nodes and establish a controllable number of semantically clear similarity relationships between nodes. This allows sentiment features to propagate along propagation paths within the graph structure according to underlying semantic logic, expression style, and social attributes. The system's graph construction module can dynamically adjust the number of nodes, edge density, and similarity calculation method based on the task scale to achieve efficient and stable graph representation learning.

[0039] In this implementation, each multimodal sample is represented as a single node in the graph, and the initial representation vector of the node is the fused feature vector h obtained in the previous module. i To ensure the graph structure can represent the potential semantic relationships between samples, the system first calculates the Euclidean distance between all nodes. The calculation of Euclidean distance has different performance requirements on datasets of different sizes; therefore, the system adopts a batch processing strategy, dividing the node feature matrix into blocks to reduce the memory overhead of a single computation. In a GPU environment, the system uses parallel matrix operations to accelerate distance calculation. Distance matrix d ij Once formed, the system passes it as input to the RBF kernel function module.

[0040] During the RBF kernel similarity mapping process, the system automatically estimates the kernel width parameter σ based on the distribution of the node feature matrix, ensuring a suitable similarity range across different datasets. This implementation uses the mean or median of the distance between samples as the initial value of σ, and dynamically fine-tunes it during model training to adapt to specific tasks. The RBF module outputs a similarity matrix with values ​​ranging from 0 to 1. Higher similarity values ​​indicate greater similarity between the two nodes in the multimodal feature space, and these edges will receive higher attention or propagation strength in subsequent GCN propagation.

[0041] In addition to modal feature-based similarity, this implementation introduces social attribute similarity modeling to enhance the graph structure's ability to express individual differences. The method for constructing social attribute similarity varies depending on the nature of the attribute. For discrete attributes, such as gender, cultural category, and education level, the system uses an indicator function or Euclidean distance after one-hot mapping to determine the similarity between two nodes. If the attributes are the same, a higher similarity is assigned; if the attributes are different, a lower or zero similarity is assigned. For continuous attributes, such as age, income level, or usage frequency, the system uses an exponential decay function to calculate similarity, ensuring that the smaller the attribute difference, the higher the similarity, thus creating a natural transition.

[0042] Social attributes may encompass multiple dimensions. To avoid any single dimension excessively influencing similarity fusion, this implementation normalizes all attribute similarity matrices. Based on this, the system linearly combines the attribute similarity matrices using a set of learnable or configurable attribute weights to obtain a comprehensive attribute similarity matrix S. (attr) The attribute weights can be set manually by the user, such as to emphasize the impact of a certain attribute on emotional expression; or the system can learn automatically during the training phase to ensure that the attribute weights achieve optimal results in the final prediction task.

[0043] In the feature similarity matrix S and the attribute similarity matrix S (attr) After both are constructed, this implementation method fuses them in a weighted manner to form the final multimodal-social attribute joint adjacency matrix A. The selection of the fusion weight parameter λ can be determined through experimental verification or automatic parameter tuning. This implementation method provides two weight modes: static weight mode and dynamic weight mode. In the static mode, λ is fixed before training, while in the dynamic mode, the system allows the system to gradually adjust during training, enabling the model to automatically optimize based on error feedback and achieve higher generalization ability.

[0044] After constructing the adjacency matrix, this implementation further performs sparsification. Sparsification improves the system's training efficiency and reduces meaningless propagation paths. The system employs a threshold pruning strategy, setting the weights of edges with similarity below the preset threshold to zero while maintaining the connectivity of the graph structure. To ensure the graph retains sufficient information propagation capability, the system also supports constructing a sparse graph by selecting the top-k similar nodes, thereby avoiding the loss of important connections due to sparsification.

[0045] The sparsed adjacency matrix needs to be normalized before being input into the graph neural network. This implementation uses a symmetric normalization strategy, calculated by the degree matrix D. ii =∑ j A ij And generate a normalized adjacency matrix. This normalization method ensures that nodes receive balanced and stable information from their neighborhoods during graph convolution, avoiding propagation bias caused by nodes with higher degrees.

[0046] In actual deployment, social attributes may be missing or incomplete. This implementation introduces a placeholder mechanism for missing attributes during the graph structure construction phase. For missing discrete attributes, the system assigns an independent placeholder to each node; for missing continuous attributes, it uses mean imputation or statistical distribution sampling of the dataset to generate estimated values, enabling the graph structure construction process to continue even with incomplete attributes and maintaining the graph's validity.

[0047] This implementation constructs a heterogeneous graph structure that can adapt to complex sentiment computing scenarios by jointly modeling feature similarity and social attribute similarity. This graph structure not only expresses the similarity of sentiment features between samples but also reflects the expression differences between different user groups due to social attributes, making the connections between nodes more semantic and interpretable, thus providing a solid foundation for subsequent graph neural network propagation.

[0048] After the graph structure is constructed, this implementation enters the local feature propagation stage based on Graph Convolutional Networks (GCN). The core objective of this stage is to utilize the constructed weighted adjacency matrix to enable nodes to exchange sentiment-related features within their local neighborhoods, thereby enhancing the contextuality and social attribute consistency of node representations. The GCN module aggregates and transforms node features layer by layer, enabling nodes to absorb feature patterns from similar nodes while maintaining their own modal information, thus achieving implicit modeling of sentiment expression styles, language habits, and visual behavior patterns.

[0049] This implementation uses a two- to three-layer GCN structure as the default configuration for the local propagation module. The first-layer GCN is mainly used to integrate the original modal features of the neighborhood, enabling nodes to initially obtain emotional cues from similar nodes. The second and subsequent layers of GCN continue to spread features over a wider range, allowing nodes to further perceive higher-order structural relationships through "neighborhoods of neighbors." In the specific implementation, each layer of the GCN achieves feature propagation through matrix operations, calculated as follows: in Let W be the normalized adjacency matrix, and let W be the input feature matrix of the l-th layer. (l) This is a trainable weight matrix.

[0050] Since graph convolution is highly dependent on the adjacency matrix, this implementation ensures that the input matrix is ​​properly configured before performing feature propagation. The sparsity and symmetry of the matrix are used to match the characteristics of sparse matrix multiplication, thereby improving the training efficiency of the system. In engineering deployment, to avoid the huge computational cost of full matrix multiplication, this implementation adopts sparse matrix representation and uses CSR (Compressed Sparse Row) or CSC (Compressed Sparse Column) format to store the adjacency matrix, which greatly reduces invalid storage. It also combines the GPU's native sparse operation library to accelerate the multiplication operation between the adjacency matrix and the feature matrix.

[0051] During propagation, the features of each node are normalized and weighted within their neighborhood. Due to the normalized adjacency matrix, the neighborhood information received by each node is degree-balanced, thus preventing excessive influence from highly connected nodes. To further improve training stability, this implementation adds layer normalization or batch normalization after each graph convolution layer to reduce fluctuations in activation value distribution and improve the training performance of deep GCN structures.

[0052] To enhance the expressive power of GCN within local structures, this implementation introduces a non-linear activation function after feature aggregation. ReLU activation is used by default, but the system also supports other activation functions such as GELU and LeakyReLU to adapt to the characteristics of different datasets. Simultaneously, Dropout or DropEdge strategies are incorporated into the GCN layer, causing the model to randomly ignore some connections during training, thereby reducing the risk of overfitting and improving generalization performance. Under the DropEdge strategy, the system randomly removes some edges from the graph with a certain probability, making the GCN aggregation process more robust, especially suitable for sentiment computing scenarios where social attributes are noisy or modal features are unstable.

[0053] This implementation utilizes the first-level propagation of GCN to enable nodes to receive feature supplements from their nearest neighbors. In emotional video data, nodes with similar facial expressions or emotional tones will propagate relevant features to each other. For text modalities, nodes with similar vocabulary habits or expressions will also enhance each other's semantic features, thereby generating richer local emotional contexts. At the social attribute level, nodes of users of the same age or gender will have stronger connections due to their higher similarity, and GCN will make the emotional representations of these nodes more consistent, reflecting more authentic differences in expression styles.

[0054] The second-layer GCN further expands the propagation range, enabling nodes to indirectly access information from their second-order neighbors. When two nodes differ significantly in their original modal features, but their common neighbors express similar emotional patterns, the second-layer GCN will allow them to complement each other to some extent. In sentiment computing tasks, this second-order propagation often helps capture more complex emotional information, such as cross-sentence emotional coherence, long-term semantic dependencies, or potential facial action combinations.

[0055] In practical engineering implementations, the weight matrix in GCN is randomly initialized and continuously updated during the training phase using gradient descent. This implementation uses Xavier or Kaiming initialization by default to ensure that features have an appropriate numerical range in the early stages of propagation, avoiding gradient explosion or decay. During training, the system calculates gradients through backpropagation and uses the AdamW optimizer to update the parameters of the GCN module, thereby improving the model's ability to capture local feature differences.

[0056] In scenarios with a large number of nodes or where the graph structure changes dynamically over time, such as real-time emotion monitoring or multi-person dialogue systems, this implementation supports a mini-batch graph training strategy. The system automatically extracts subgraphs based on the topological relationships of the nodes and performs GCN operations on the subgraphs, thereby effectively reducing memory usage during training and enabling the model to run in resource-constrained environments.

[0057] After propagation through multiple layers of GCN, each node will possess a local deep representation that combines multimodal features, neighborhood emotion patterns, and social attribute relevance. These features will serve as input for global modeling in subsequent modules, enabling the model to capture more complex and distant dependencies and achieve accurate predictions in the final emotion recognition task.

[0058] Before entering the Transformer model, this implementation first performs serialization processing on the node features from the GCN output. Specifically, the system arranges all nodes in the graph into a fixed-length sequence according to their index or sample order, so that each node corresponds to a token in the sequence. To eliminate ambiguity in the position of nodes in the sequence, this implementation adds a trainable position vector to each node during the serialization process, enabling the Transformer to distinguish the logical positions of different nodes in the input sequence.

[0059] This approach utilizes a multi-head self-attention structure to capture various types of sentiment relevance in parallel. The calculation of each attention head includes the formation process of the query vector (Q), key vector (K), and value vector (V). The node feature vectors are linearly mapped and used as Q, K, and V, respectively, to calculate the attention weights between nodes.

[0060] In the standard Transformer architecture, the attention weights are calculated by... The decision was made to add a graph structure bias term to this, so that the model can simultaneously consider the topological information of nodes in the graph structure. The normalized adjacency matrix is ​​mapped to attention bias, and this bias is added when calculating attention weights, thereby giving higher attention weights to similar nodes, so that distant but highly related nodes can get enough expression opportunities in self-attention.

[0061] In practical engineering implementation, a masked softmax strategy is adopted for the attention weight matrix to maintain the numerical stability of the model under large-scale node input. For node pairs with zero edge weights in the adjacency matrix, the system can enable attention mask as needed, allowing the Transformer to automatically filter out information interference between irrelevant nodes, thereby improving the sparsity of the attention distribution and enhancing the interpretability of the model.

[0062] Attention-weighted node features undergo linear projection and residual connection operations. Layer normalization is added after each attention module to prevent uncontrolled changes in feature distribution within deep networks. Simultaneously, the residual structure ensures good gradient propagation capabilities even with increased depth, reducing the vanishing and exploding problems that may occur in deep Transformers during training.

[0063] In addition to the self-attention sublayer, the Transformer module also includes a feedforward neural network sublayer. This implementation employs a two-layer feedforward structure, expanding the hidden dimensions within the network to enable the Transformer to perform non-linear transformations on node features and enhance the model's expressive power. The GELU activation function is used in the FFN sublayer to improve the fitting ability of features to capture subtle tone changes, slight facial expressions, or emotional cues caused by semantic ambiguity in scenarios with weak emotional shifts.

[0064] To improve the efficiency of global feature modeling, this implementation supports a variable-depth Transformer architecture. A 4-layer encoder is used by default, but can be automatically expanded to 6 or 8 layers depending on available computing power when higher accuracy or more complex emotion patterns are required. During deployment, to adapt to mobile devices, this implementation supports depth pruning. By automatically evaluating the importance of each layer to the final prediction result, key layers are retained and redundant layers are removed, achieving lightweight inference.

[0065] After global modeling using multiple Transformer layers, the feature vector of each node has incorporated information from multimodal inputs, neighborhood propagation, and the global semantic structure. This implementation uses the node representations output by the Transformer as input for the final emotion prediction, enabling the model to integrate local and global features of emotion expression at a higher level, achieving accurate and stable emotion recognition results.

[0066] The aggregated features are then input into the classifier or regressor module. The classifier module typically includes one or more fully connected layers, which transform high-dimensional emotional features into emotional categories or emotional labels through nonlinear mapping. This implementation supports various tasks such as traditional discrete emotion classification (e.g., happiness, anger, sadness), binary emotion classification (e.g., positive / negative), and psychological state recognition (e.g., depression level classification). For continuous emotional dimensions (e.g., emotional valence and arousal), the system uses the regressor module to predict numerical labels, supporting single-task regression and multi-output regression modes.

[0067] To enhance the model's predictive ability, this implementation introduces feature regularization and semantic consistency constraints. Regarding feature regularization, the system applies regularization terms or graph smoothing constraints to node representations, ensuring that node representations maintain appropriate compactness in high-dimensional space and preventing overly discrete feature distributions from affecting the stability of the classification boundary. Regarding semantic consistency, the system applies similarity constraints to the prediction distributions among samples, making the prediction results for similar samples more similar, thereby improving the model's discriminative ability and robustness.

[0068] A key innovation of this implementation lies in the introduction of social attribute constraint loss. This loss aims to encourage nodes with similar social attributes to remain close in the representation space, enabling the model to automatically learn differences in user expression styles caused by social factors. The social attribute constraint loss is calculated as follows: for each pair of nodes, the distances between the nodes are weighted and summed according to the weights in their attribute similarity matrix, ensuring that the representations of similar nodes converge in the same direction during training. This module not only improves prediction accuracy but also enhances the model's interpretability, making it more transparent in user-dependent sentiment calculation tasks.

[0069] In addition to social attribute constraints, this implementation also introduces a graph structure consistency loss. In this loss, the system requires that the node representations output by the Transformer should be consistent with the connection relationships in its graph structure. When two nodes have high similarity in the graph structure, their distance in the embedding space should be small; conversely, when the similarity is low, the distance between the node representations should be large.

[0070] For sentiment prediction tasks, this implementation uses cross-entropy loss, mean squared error loss (MSE), Huber loss, or a combination thereof for training. In classification tasks, cross-entropy loss measures the difference between the model's output probability distribution and the true label distribution; in regression tasks, mean squared error loss learns the continuous changes in sentiment intensity, while Huber loss provides better robustness to outliers or unstable labels. The system automatically selects the most suitable combination of loss functions based on the task type and data characteristics.

[0071] The total loss in this implementation is a weighted combination of multiple sub-losses, including classification loss or regression loss, social attribute constraint loss, etc. Figure 1 Consistency loss, regularization loss, and other auxiliary losses. The total loss formula is as follows: Here, λ is a hyperparameter that the system can automatically optimize during training based on the validation set results. This approach achieves a balance between steady-state learning, cross-modal consistency, social attribute understanding, and global semantic modeling by integrating multiple loss terms.

[0072] After training, this implementation method integrates the optimized weight parameters, graph structure construction strategy, and feature normalization method into the deployed model. This allows the system to directly receive multimodal inputs from real-world scenarios during the inference phase and generate emotion results through the same feature extraction and prediction process. Because the loss design includes graph structure consistency terms and social attribute constraints, the deployed model maintains stable and consistent outputs across different user groups, expression styles, and modal qualities, thus achieving reliable identification of the target emotional state. This section concludes here, laying the foundation for subsequent model training strategies and deployment mechanisms.

Claims

1. A multi-modal depression recognition method fusing social attribute graphs, characterized in that, Comprising the following steps: (1) obtaining a face image sequence, a speech signal and text data of a subject to be tested; (2) encoding the social attribute information of the subject, mapping attributes such as gender, age and occupation to node features, and constructing a social relationship graph based on attribute similarity; (3) using a graph neural network to propagate messages on the social relationship graph to generate social attribute graph features; (4) using visual Transformer, speech sequence model and text encoding model to extract multi-modal emotion features respectively; (5) embedding the social attribute graph features into a multi-modal Transformer network, and realizing joint modeling of image, speech and text features through a cross-modal attention mechanism; (6) training the fused features based on a joint optimization strategy, while minimizing the emotion classification error and the social similarity regularization term; (7) outputting the depression recognition result of the subject.

2. The system of claim 1, wherein the modal feature processing module comprises: a text preprocessing unit for word segmentation, denoising and context encoding of the text; a speech feature extraction unit for extracting acoustic features such as MFCC and energy envelope from the speech; a video frame processing unit for extracting feature signals such as facial key points and expression intensity from the video.

3. The system of claim 1, wherein the social attribute graph construction module comprises: (1) a feature encoding unit for vectorizing the user's social attributes; (2) a similarity measurement unit for calculating the attribute similarity between samples, determining the edge weight between nodes by Euclidean distance, cosine similarity or RBF kernel function; (3) a graph generation unit for constructing a social attribute graph containing nodes, edge weights and attribute features.

4. The system of claim 3, wherein, The formula for calculating the edge weight in the similarity measurement unit is: where a i , a j is the social attribute vector of user i and user j, and σ is a scale adjustment parameter.

5. The Graph-Transformer feature fusion module comprises: a graph convolution unit for updating node features and capturing local neighborhood dependencies on the social attribute graph; a cross-modal attention unit for multi-head self-attention feature interaction between text, speech and video modalities; a graph-Transformer joint encoder for fusing graph convolution output and cross-modal attention output to realize joint modeling of multi-modal features and social information.

6. The system of claim 5, wherein the graph convolution unit adopts a multi-layer graph convolution structure, and each layer performs: H (l+1) = σ(D -1 / 2 AD -1 / 2 H (l) W (l) ) where A is the graph adjacency matrix, D is the degree matrix, H (l) is the feature of the i-th layer node, W (l) is the learnable weight matrix.

7. The system of claim 5, wherein the cross-modal attention unit realizes feature interaction between modalities through a Query, Key, Value three-branch structure, and the steps include: (1) projecting each modality feature to a unified dimension; (2) calculating the inter-modal attention matrix; (3) weighting and fusing the modality features according to the attention weights to form a cross-modal relationship representation.

8. The system of claim 1, wherein the emotion prediction module comprises: a task loss calculation unit for calculating emotion classification or emotion intensity regression error; a social similarity regularization unit for constraining the emotion prediction results of similar users in the social attribute graph to be closer. The joint loss optimization unit is configured to combine the task loss and the similarity regular term in a preset proportion for model training.

9. The system of claim 8, wherein, The total loss function of the joint loss is: wherein is a task loss, is an attribute consistency loss, is a graph structure constraint loss, and a, b are weight coefficients.

10. The system of any one of claims 1 to 9, wherein, The data storage analysis module is configured to archive the user historical modal data, the graph structure information, the prediction result and the model intermediate layer representation, so as to support model backtracking analysis, individual monitoring and system performance evaluation.

Images