Emotional state evaluation method based on multi-modal data

Abstract

The invention relates to the technical field of artificial intelligence and emotion calculation, and discloses an emotion state evaluation method based on multi-modal data, and the method comprises the steps: obtaining and decoupling an implicit signal, an explicit signal and situation data of an evaluated object; and respectively generating implicit features, explicit features and context vectors through double-flow decoupling coding and context knowledge graph coding. And the core engine determines the internal emotional state and the extrinsic emotional representation in parallel, and dynamically regulates and controls a bridging conversion process from the internal state to the predicted extrinsic representation by utilizing the situation vector. Finally, the emotion regulation intensity is calculated by comparing actual and predicted extrinsic representations, and the emotion regulation intensity, the internal state and the extrinsic representations are jointly used as a multi-dimensional evaluation result to be output. According to the method, the dynamic relationship and the adjustment strategy between the internal feeling and the external expression can be disclosed, a deeper and more three-dimensional analysis view angle is provided for the fields of mental health, human-computer interaction and the like, and the method has remarkable application value.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and affective computing, specifically to a method for assessing emotional states based on multimodal data. Background Technology

[0002] Affective computing, as an important branch of artificial intelligence, aims to endow machines with the ability to understand, recognize, and respond to human emotions, demonstrating enormous application potential in fields such as mental health assessment, human-computer interaction, education, and market analysis. Currently, automated emotion state assessment technologies primarily rely on the analysis of an individual's external behavioral performance, such as analyzing facial micro-expressions and body postures through computer vision technology, or analyzing tone, speech rate, and acoustic features through speech signal processing technology. Some more advanced technologies attempt to integrate data from multiple modalities, such as simultaneously processing video and audio signals, in order to obtain more reliable emotion recognition results than those from a single modality.

[0003] However, existing technological solutions generally suffer from a fundamental limitation: they tend to directly equate observed external emotional expressions with an individual's true internal emotional state. This approach implicitly assumes that external behavior is a faithful and direct reflection of inner feelings. But in the complex and ever-changing real-world social environment, this assumption often fails. As social beings, humans' emotional expressions are heavily constrained by social norms, cultural background, communication intentions, and specific situations. For example, in formal business negotiations, individuals may suppress their anxiety and project confidence and composure; in the service industry, employees are required to maintain a positive and friendly attitude even when mentally exhausted. Existing technologies neglect this potential separation between inner feelings and external expressions, leading to biased and superficial assessments. They are often "situationally blind" when processing signals, unable to consider the current social context, and therefore unable to understand the drastically different psychological meanings that the same smile might convey in a friend's gathering versus a work report. Due to this lack of understanding of the context, existing technologies naturally cannot model and quantify the core psychological process of "emotion regulation." Emotion regulation, the process by which an individual actively manages their emotional expression in order to achieve a specific goal, is a key indicator for measuring an individual's psychological maturity and social adaptability. However, existing assessment methods lack depth in this area of ​​analysis.

[0004] Furthermore, even multimodal systems that simultaneously employ physiological signals (such as EEG and ESC) and behavioral signals often employ relatively rudimentary data fusion strategies. They typically use simple feature splicing or decision-level weighted fusion, failing to capture the complex, nonlinear, and temporally delayed deep connections between implicit physiological responses and overt behavioral expressions. In summary, current technologies have significant shortcomings in distinguishing between internal states and external expressions, fusing contextual information, and quantifying emotion regulation behaviors, urgently requiring a new technological solution that can provide deeper and more comprehensive insights into emotional states. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an emotion state assessment method based on multimodal data. The technical problem to be solved is that existing emotion assessment technologies usually treat internal emotions and external expressions as equal and ignore the key influence of social context, resulting in the inability to quantify and assess an individual's emotion regulation behavior.

[0006] To achieve the above objectives, the present invention provides the following technical solution: an emotion state assessment method based on multimodal data. This method decouples emotional information into two levels, internal and external, and explicitly models the conversion relationship between the two. At the same time, it introduces contextual information for dynamic adjustment, thereby achieving a three-dimensional and in-depth analysis of emotional states.

[0007] Specifically, the present invention provides a method for assessing emotional state based on multimodal data, comprising the following steps: First, multimodal data is acquired and decoupled into implicit and explicit signals. This step aims to separate two types of information channels with different properties from the raw data stream. Implicit signals tend to reflect an individual's unconscious, genuine physiological and psychological reactions. In one specific embodiment, these may include physiological signals such as electroencephalogram (EEG) and electrodermal activity (EDA) and / or micro-expression signals from the facial region. Explicit signals, on the other hand, tend to reflect signals that are subjectively controlled by an individual for social communication. In one specific embodiment, these may include text content, speech signals, and / or macro-expression signals from the face during interaction.

[0008] Subsequently, based on the implicit signal, the underlying emotional state is determined. This step involves encoding the implicit signal using a temporal network model (such as a temporal convolutional network or a recurrent neural network) specifically designed for processing temporal information, extracting implicit features that reflect its dynamic changes. Then, based on these implicit features, a recursive state update mechanism (such as a gated recurrent unit) is used to continuously track and determine the underlying emotional state.

[0009] Simultaneously, based on the explicit signals, explicit emotion representations are determined. In one embodiment, this step encodes the explicit signals using a multimodal fusion model (such as a multimodal Transformer) capable of effectively processing and fusing multiple data sources to capture their rich semantics and cross-modal associations, thereby obtaining explicit features. Then, the explicit features are converted into the explicit emotion representations using a mapping network (such as a multilayer perceptron).

[0010] One of the core innovations of this invention lies in the fact that, after determining the two states mentioned above, the system predicts the overt emotional representation based on the internal emotional state, thereby obtaining a predicted overt emotional representation. This step aims to explicitly model the transformation process from "inner feelings" to "external expression".

[0011] To make the modeling of this conversion process more closely resemble real-world interactions, this invention also introduces contextual factors. In a preferred embodiment, before predicting the explicit emotional representation, the invention further includes acquiring contextual data, such as social or environmental contextual data, related to the multimodal data, and generating a contextual vector based on the contextual data to modulate the prediction process. For example, the contextual data (such as "office" or "superior") can be constructed as a contextual knowledge graph, and a graph neural network can be used to encode the contextual knowledge graph to obtain the contextual vector.

[0012] Furthermore, the step of predicting the explicit emotional representation is specifically performed by a bridging module dynamically adjusted by contextual information. This bridging module receives the intrinsic emotional state as input, and its internal computation process is dynamically gated by the context vector. In one specific embodiment, the dynamic gating generates a set of modulation parameters (such as scaling factors) by inputting the context vector into a gating network. and bias factor The modulation parameters are then used to linearly modulate the features within the bridging module. For example, for the input features of a certain computational layer in the bridging module... Its modulated output characteristics It can be calculated using the following formula: ; in, This represents element-wise multiplication. In this way, the bridging module can learn different mapping rules from internal state to external expression based on different contexts, and finally output the predicted explicit emotional representation.

[0013] Finally, the intensity of emotion regulation is determined based on the difference between the explicit emotional representation and the predicted explicit emotional representation. This intensity of emotion regulation is a novel metric that quantifies the extent to which an individual masks or modulates their inner feelings. In one embodiment, both the explicit emotional representation and the predicted explicit emotional representation are probability distributions, and the intensity of emotion regulation... The KL divergence between the two can be calculated to obtain: ; in, This represents the probability distribution of the actual explicit emotional representation. This represents the probability distribution of the predicted explicit emotional representation. Finally, the system outputs a multidimensional assessment result that includes the internal emotional state, the explicit emotional representation, and the intensity of emotion regulation, thus providing a comprehensive emotional insight.

[0014] To train the method of this invention, an end-to-end optimization can be performed using a joint loss function. In one embodiment, the joint loss function comprises three parts: an intrinsic state loss for supervising the learning of the intrinsic emotional state, an explicit representation loss for supervising the learning of the explicit emotional representation, and a bridging prediction loss for supervising the learning of the predicted explicit emotional representation. By weighted summing and minimizing these three parts of the loss, the various modules of the model can be synergistically optimized.

[0015] This invention provides a method for assessing emotional states based on multimodal data. It has the following beneficial effects: 1. This invention transcends the limitations of traditional emotion recognition, which only outputs a single emotion label. It provides a multi-dimensional assessment result that includes internal emotional state, overt emotional representation, and the intensity of emotion regulation. This is achieved through the system's decoupling of implicit and overt signals and the final quantification of their differences. In this way, the assessment result is no longer a flat "happiness" or "sadness," but is transformed into a three-dimensional, layered portrait of psychological state, greatly enhancing the depth and information richness of the assessment result.

[0016] 2. This invention effectively reveals the inconsistency between an individual's inner feelings and outward expression, commonly referred to as "emotional masquerade" or "emotional suppression." The key to achieving this effect lies in the parallel establishment of independent inner emotional state tracking pathways and outward emotional representation mapping pathways within the technical solution. By independently modeling these two states, this invention can capture and ultimately quantify the degree of deviation between them, thus providing a reliable technical means for identifying complex psychological activities that were previously difficult to detect, such as social masquerade and emotional concealment under stress.

[0017] 3. This invention significantly improves the accuracy and real-world adaptability of emotion assessment because it is the first to explicitly model social context as a key variable influencing emotion expression. By introducing a contextual knowledge graph encoding module, the system can understand the semantic differences between different scenarios such as "meeting room" and "coffee shop," and dynamically adjust the mapping function from internal state to external expression using a feature-level linear modulation mechanism in the core assessment and transformation engine. This makes the assessment no longer a rigid pattern matching, but rather possesses human-like intelligence that flexibly judges different social environments.

[0018] 4. This invention provides an objective and quantifiable indicator for assessing emotion regulation behavior, filling a gap in existing technologies. By using a bridging module that predicts "natural" overt expressions based on internal states and context, and comparing the predicted results with actually observed overt expressions, this invention creatively transforms the abstract psychological concept of "emotion regulation" into a concrete and calculable numerical measure of emotion regulation intensity. This indicator provides data support for applications such as mental health screening, social skills training, and monitoring of the state of high-pressure occupational groups.

[0019] 5. The joint training framework of this invention not only improves the overall performance of the model but also enhances the interpretability of its evaluation results. By designing a joint loss function that includes intrinsic state loss, explicit representation loss, and bridging prediction loss, the model is guided to learn the three sub-tasks of intrinsic tracking, extrinsic recognition, and contextual transformation in a structured manner. This decompositional learning paradigm avoids the ambiguity of the internal decision-making process of traditional end-to-end models, allowing researchers to examine the output of each stage and thus more clearly understand the basis of the model's judgments. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the functional modules of the emotional state assessment system of the present invention; Figure 2 This is a schematic diagram of the internal logic of the core evaluation and conversion engine of this invention; Figure 3 This is a schematic diagram of the contextual knowledge graph encoding process of the present invention; Figure 4 This is a schematic diagram of the joint training process of the model according to the present invention. Detailed Implementation

[0021] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Please see the appendix Figure 1 To be continued Figure 4 This invention provides a method for assessing emotional states based on multimodal data, which is executed on a computing device equipped with a processor and memory. The system aims to provide a comprehensive, high-dimensional assessment of emotional states by deeply analyzing the internal and external layers of human emotions and understanding the dynamic strategies of emotional expression in conjunction with contextual factors.

[0023] The methodology for implementing an emotional state assessment system may include the following steps: First, the data acquisition and preprocessing module collects raw data streams reflecting the state of the evaluated object. After preprocessing the collected data, the module decouples it into three independent data streams: implicit signals, which tend to reflect an individual's unconscious physiological reactions; explicit signals, which tend to reflect an individual's subjectively regulated social expressions; and contextual data, which describes the external environment and social context in which the emotional event occurs.

[0024] Next, the implicit and explicit signals are fed into a dual-stream decoupled feature coding module. This module contains parallel processing paths with different structures, encoding the implicit and explicit signals separately to extract their deep feature representations and output implicit and explicit feature sequences. This decoupled coding design aims to separate the different emotional cues carried by the two types of signals at the source.

[0025] Meanwhile, the contextual data is fed into the contextual knowledge graph encoding module. This module is responsible for transforming unstructured contextual information into structured graph data and using graph representation learning techniques to encode it into a fixed-dimensional contextual vector that condenses information about the current social environment.

[0026] Subsequently, the implicit feature sequence, explicit feature sequence, and context vector generated in the preceding steps are input together into the core evaluation and transformation engine. This engine is the core component for achieving the technical effects of this invention, and it performs three parallel key tasks internally. First, based on the implicit feature sequence, it tracks and determines a continuously changing internal emotional state. Second, based on the explicit feature sequence, it maps and determines an explicit emotional representation at a specific moment. Third, and also one of the key technical points of this invention, the engine uses the internal emotional state as input and receives the context vector as a dynamic adjustment signal, generating a predicted explicit emotional representation through an internal bridging transformation mechanism.

[0027] Finally, the results generation and model training module receives output from the core evaluation and transformation engine, including the determined internal emotional state, the determined explicit emotional representation, and the generated predicted explicit emotional representation. Based on the quantitative difference between the explicit and predicted explicit emotional representations, the results generation and model training module calculates a novel metric, namely, emotion regulation strength. Ultimately, the system integrates the internal emotional state, explicit emotional representation, and emotion regulation strength as the final multidimensional evaluation result. During the training phase, the results generation and model training module is also responsible for calculating the joint loss to optimize the entire system end-to-end.

[0028] In embodiments of the present invention, the data preparation stage of the emotional state assessment method is defined in detail. This stage is performed by the data acquisition and preprocessing module, whose core task is to collect multi-source heterogeneous raw data and transform it into a clean and structured input format suitable for deep learning model processing.

[0029] The method begins with a data acquisition step. Implicit signals are acquired using non-invasive sensing devices. For example, multi-channel electroencephalography (EEG) signals are acquired by wearing an EEG cap equipped with multiple electrodes; simultaneously, electrical activity on the skin (EDA) signals are acquired using sensors worn on the fingertips or wrists. Explicit signals are acquired using standard audio and video recording equipment. For example, the speech stream of the subject during the interaction is captured using a directional microphone, and a video stream of their facial area is recorded using a high-definition camera. Contextual data is entered as text labels through a human-computer interface to describe the specific scenario in which the interaction occurs, such as "reporting work to a superior in the office."

[0030] After acquiring the raw data stream, the data acquisition and preprocessing module performs a series of targeted preprocessing operations. For EEG signals, a bandpass filter is first applied to remove power frequency noise and low-frequency baseline drift. Next, independent component analysis (ICA) is used to identify and remove common physiological artifacts caused by eye movements, blinking, or muscle activity, thereby purifying neural oscillation signals related to cognitive and emotional activities.

[0031] For EDA signals, the preprocessing aims to decompose them into a tonic component reflecting the background arousal level and a pulsed component reflecting the transient response to a specific stimulus. This decomposition process is implemented using a convex optimization algorithm to decompose the original signal into two mutually orthogonal components for subsequent model use.

[0032] For audio streams, preprocessing begins by applying spectral subtraction or a similar algorithm to suppress ambient background noise. Subsequently, a speech activity detection (VAD) algorithm is used to accurately segment out valid segments containing human speech. For each segmented speech segment, the system converts it into a time-frequency representation, such as a Mel-spectrogram, which visually displays the energy distribution of the speech signal in time and frequency.

[0033] For facial video streams, the preprocessing involves frame-by-frame processing. On each frame, a face detection algorithm is first run to locate facial regions. After localization, a facial keypoint detection model calibrates the precise coordinates of areas such as the eyes, eyebrows, nose, and mouth. Based on these keypoint coordinates, the system performs affine transformations on all facial images, achieving rotation, scaling, and alignment to ensure all faces are in similar, normalized poses. Furthermore, techniques such as histogram equalization are applied to mitigate the impact of illumination variations on image features.

[0034] After all preprocessing is complete, the method performs the final data alignment and segmentation steps. The cleaned data streams from different modalities are aligned along the timeline according to their respective timestamps. Then, the aligned multimodal data streams are synchronously segmented into multiple data segments in preset fixed-length units (e.g., several seconds). Each data segment constitutes a complete input sample for a single processing iteration of the model, containing the timing information of implicit and explicit signals occurring in parallel within that time period. This segmentation method preserves the natural occurrence delays between different modal signals, eliminating the need for mandatory frame-level strict synchronization.

[0035] After the data acquisition and preprocessing module outputs clean, aligned data segments, the method is then executed by the dual-stream decoupled feature encoding module. The core function of this module is to extract deep features from the implicit and explicit signals respectively through two parallel encoders that are distinct in structure and function, in order to generate feature sequences that retain their unique information dimensions.

[0036] For processing implicit signals, this embodiment employs a temporal convolutional network (TCN) as the implicit stream encoder. The encoder's design fully considers the long-term temporal dependence characteristics typically exhibited by physiological signals (such as EEG and EDA). Its network structure consists of a series of stacked residual blocks, each containing two layers of dilated causal convolutions. The causal convolution design ensures that processing occurs at each time step... When dealing with signals, only current and past information is used, preventing the leakage of future information, which aligns with the inherent logic of time series prediction. Dilated convolution, through an exponentially increasing dilation factor, allows upper-layer networks to obtain a large receptive field with fewer parameters, thus efficiently capturing slowly changing physiological rhythms or trends throughout the entire time window. After processing by this implicit flow encoder, the input implicit signal data segment is transformed into a high-dimensional implicit feature sequence.

[0037] For the processing of the external signal, considering the complex and nonlinear synchronization and complementarity between speech and facial video, this embodiment adopts a Transformer-based multimodal fusion architecture as the external stream encoder. This encoder first uses two independent, modality-specific shallow convolutional neural networks to perform preliminary spatial feature extraction on the input Mel spectrogram sequence and facial image sequence, respectively, transforming the audio and video data at each time step into initial feature embeddings.

[0038] Subsequently, these initial feature embedding sequences are fed into a multi-layered, stacked cross-modal Transformer encoder. Within each Transformer layer, self-attention operates independently within its respective modality, enabling the model to capture temporal dynamics within that modality—for example, the complete evolution of a facial expression from its inception to its peak and then to its decay, or the intonation of speech. Next, cross-modal attention plays a crucial role in fusion. In this mechanism, feature sequences from one modality (e.g., video) act as queries, while feature sequences from another modality (e.g., audio) act as keys and values. This process allows video features to proactively "query" and aggregate the most relevant audio features within the same time window, and vice versa. Through this deep interaction and fusion across multiple layers, the encoder learns deep associations beyond simple splicing, such as the subtle coupling between specific smiling muscle movements and corresponding vocal timbre. Finally, the explicit stream encoder outputs a unified explicit feature sequence that fully integrates audio and video information. The method includes a specific set of steps performed by a contextual knowledge graph encoding module. This module's function is to transform discrete, symbolic contextual data into a continuous, machine-understandable vector representation, namely, a contextual vector.

[0039] The execution of this step begins with the structured processing of contextual data. In this embodiment, the system pre-constructs an ontology knowledge base containing general social concepts. This knowledge base is organized in the form of a graph, where nodes represent entities (such as "person," "location," and "event"), and edges represent relationships between entities (such as "located in," "participating," and "reporting to"). When a contextual text label describing a specific scenario (e.g., "reporting project progress to team members in the meeting room") is input, a natural language processing unit first performs entity recognition and relationship extraction to identify key information such as (location: meeting room), (speakers: team members), and (event: reporting project).

[0040] Subsequently, these extracted entities and relationships are linked to a pre-built ontology knowledge base, dynamically generating a small-scale contextual knowledge subgraph relevant to the current specific context. This subgraph not only contains direct information about the current context but also inherits broader background knowledge from the ontology base through links; for example, "team member" is a type of "colleague," and "reporting" is a formal "communication" behavior. This structured representation method, compared to the simple bag-of-words model, can capture the inherent structure and semantics of the context more profoundly.

[0041] After generating the contextual knowledge subgraph, this embodiment employs a Graph Attention Network (GAT) as the graph encoding network to generate the final context vector. Unlike traditional graph convolutional networks, the GAT does not treat all neighbors equally when aggregating neighbor node information; instead, it introduces an attention mechanism. For any node in the graph, the network calculates the relevance weight between it and each neighbor node, and then uses this weight to perform a weighted summation of the features of the neighbor nodes to update the representation of the central node.

[0042] By stacking multiple layers of such graph attention layers, each node can progressively aggregate information from its multi-hop neighbors, thereby perceiving a wider range of graph structure. Finally, a graph-level readout function is applied, such as mean pooling or max pooling on the updated representations of all nodes, to integrate the information from the entire graph into a fixed-dimensional, dense context vector. This vector It is a highly condensed mathematical description of the current situation, capturing complex factors such as social environment, interpersonal relationships, and the nature of events, providing key conditional inputs for subsequent dynamic gating adjustment.

[0043] In this embodiment of the invention, the core evaluation and transformation engine receives implicit feature sequences, explicit feature sequences, and context vectors from the preceding modules, and executes the core processing steps of the method of the present invention based on these sequences. The internal operation of the engine is divided into two parallel state modeling paths and one context-controlled transformation path.

[0044] In the first state modeling path, the method tracks internal emotional states. This embodiment employs a gated recurrent unit (GRU) network to perform this task. At any time step... The GRU unit receives the implicit feature vector at the current time step as input and combines it with its feature vector at the previous time step. The maintained internal hidden state, through its internal update and reset gates, recursively calculates the new hidden state at the current moment. This new hidden state is then determined as the internal emotional state at the current moment. Because of the cyclical structure of GRU, inner emotional states can effectively integrate historical information, thereby capturing the dynamic process of how real inner feelings evolve smoothly over time.

[0045] In the parallel second state modeling path, the method maps explicit emotional representations at specific moments. This process is implemented through a multilayer perceptron (MLP) network. This network directly receives the current time step. The explicit feature vector is taken as input, passed through a series of nonlinear transformation layers, and then output as a probability distribution by a Softmax function. This probability distribution is determined as the explicit emotion representation at the current moment. It is distributed across multiple predefined emotion categories, directly reflecting the emotions displayed by the evaluated subject to the outside world at that moment.

[0046] A core technical contribution of this invention lies in its ability to not only independently determine the two states mentioned above, but also explicitly model the transformation relationship between them, which is dynamically influenced by the external context. This functionality is achieved through the engine's transformation path. First, a multilayer perceptron, acting as a gating network, receives a context vector from the contextual knowledge graph encoding module. As input, it outputs a pair of modulation parameters: a scaling vector. and a bias vector .

[0047] Subsequently, a multilayer perceptron network, acting as a bridging module, is used to determine the intrinsic emotional state. Its primary input is used to predict corresponding overt emotions. Its unique feature lies in the fact that the computational flow within this bridging module is modulated by parameters. Precise control. Specifically, for any computational layer in this bridging module, its input feature vector is denoted as... Then its output feature vector It is not solely determined by fixed network weights, but is calculated through feature-wise linear modulation as follows:

[0048] in, It is the input feature vector of the computation layer; It is the modulated output feature vector; It is a scaling vector generated by a gating network; It is a bias vector generated by a gating network; The symbol represents element-wise multiplication.

[0049] Through this mechanism, vectors from different contexts Different will be generated By combining these elements, the activation values ​​of each layer within the bridging module are dynamically altered, essentially configuring a set of context-specific "knobs" for the mapping function from internal state to external expression. This design allows the model to learn that the same internal feeling can be modulated into drastically different external expressions in different social situations (e.g., in a serious meeting or in casual conversation). The final output of this bridging module is the predicted overt emotional representation. .

[0050] In this embodiment of the invention, after the core evaluation and transformation engine completes its computational tasks, the final steps are performed by the result generation and model training module. This module has a dual responsibility: first, in the inference phase, it integrates the computational results to generate a multi-dimensional evaluation vector with rich insights; second, in the training phase, it constructs and optimizes a joint loss function to achieve end-to-end learning for the entire system.

[0051] Specifically, when generating the final evaluation result, the module first aggregates three key outputs from the core evaluation and transformation engine: the intrinsic emotional state determined by recursive updates. The actual explicit emotional representation determined by direct mapping ; and the predicted explicit emotional representations generated by the context-gated bridging module. .

[0052] A key technical aspect of this invention is that it does not stop at reporting internal and external states separately, but further quantifies the dynamic relationship between the two. This quantification is achieved by calculating a novel indicator—the intensity of emotion regulation. This is achieved through [a specific method / mechanism]. The indicator aims to measure the degree to which an individual modulates or conceals their inner feelings when expressing them externally in a specific context. In this embodiment, this strength is obtained by calculating the difference between the probability distribution of the actual overt emotional representation and the probability distribution of the predicted overt emotional representation. Specifically, it is calculated using the Kullback-Leibler Divergence:

[0053] in, Represents the KL divergence calculation function; In time step The probability distribution of actually observed overt emotional representations; and At the same time step, the model predicts the probability distribution of explicit emotional representations based on internal states and context. The larger the value, the further the actual outward expression deviates from the "natural" prediction based on inner feelings, thus indicating stronger emotion regulation behavior. Ultimately, the system will interpret the inner emotional state... Overt emotional representations and intensity of emotion regulation The results are integrated into a triple vector, which serves as a complete and multi-dimensional assessment of the emotional state at that moment.

[0054] During the training phase, in order to achieve coordinated optimization of all learnable parameters in the entire system, this embodiment employs a joint loss function. As the global training objective, the joint loss function consists of a weighted sum of three independent loss terms, each corresponding to one of the three core tasks the model needs to learn.

[0055] The first item is the intrinsic state loss. It is used to monitor the accuracy of tracking intrinsic emotional states. It calculates the intrinsic emotional states determined by the model. Compared with human-labeled real emotional state tags The mean square error (MSE) between them is used to define the relationship.

[0056] The second term is the explicit representation loss. It is used to supervise the accuracy of explicit emotion representation mapping. It calculates the explicit emotion representation determined by the model. Compared with human-labeled real emotion category tags It is defined by the cross-entropy loss between them.

[0057] The third item is the bridging prediction loss. Its purpose is to supervise the context-gated bridging module in learning the correct rules for the transition from internal to external emotions. It also calculates the explicit emotional representations predicted by the bridging module. With the category tag of true emotions The cross-entropy loss is defined as the loss between the two.

[0058] Finally, the total loss function Defined as the weighted sum of these three loss terms:

[0059] in, , and This is a non-negative hyperparameter used to balance the learning progress of different tasks. Throughout the training process, the system iteratively minimizes this joint loss function using gradient descent optimization algorithms (such as the Adam optimizer). This drives the parameters of all modules to be updated in a direction that can simultaneously and accurately track internal states, identify external expressions, and understand the context-dependent transformation relationships between them.

[0060] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for assessing emotional state based on multimodal data, characterized in that, Includes the following steps: Acquire multimodal data and decouple the multimodal data into implicit signals and explicit signals; Based on the implicit signals, the internal emotional state is determined; Based on the explicit signals, determine the explicit emotional representation; Based on the internal emotional state, predict the external emotional representation to obtain the predicted external emotional representation; Based on the difference between the explicit emotional representation and the predicted explicit emotional representation, the intensity of emotion regulation is determined, and an assessment result including the internal emotional state, the explicit emotional representation, and the intensity of emotion regulation is output.

2. The method according to claim 1, characterized in that, The implicit signals include physiological signals or micro-expression signals; the explicit signals include text signals, voice signals, or facial macro-expression signals.

3. The method according to claim 1, characterized in that, The steps for determining the inner emotional state include: The implicit signal is encoded using a temporal network model to obtain implicit features; Based on the implicit features, the intrinsic emotional state is recursively updated.

4. The method according to claim 1, characterized in that, The steps for determining the explicit emotional representation include: The explicit signal is encoded using a multimodal fusion model to obtain explicit features; The explicit features are mapped to the explicit emotion representation.

5. The method according to claim 1, characterized in that, Before predicting the explicit emotional representation, the method also includes: Acquire contextual data related to the multimodal data; Based on the context data, a context vector is generated to modulate the prediction process.

6. The method according to claim 5, characterized in that, The steps for generating the context vector include: The contextual data is constructed into a contextual knowledge graph; The contextual knowledge graph is encoded using a graph neural network to obtain the contextual vector.

7. The method according to claim 5, characterized in that, The specific steps for predicting the explicit emotional representation are as follows: Dynamic gating of a bridging module is performed using the aforementioned context vector; The internal emotional state is converted into the predicted external emotional representation through the dynamically gated bridging module.

8. The method according to claim 7, characterized in that, The dynamic gating specifically refers to: The context vector is input into a gating network to generate modulation parameters; The internal characteristics of the bridging module are linearly modulated using the modulation parameters.

9. The method according to claim 1, characterized in that, The specific steps for determining the intensity of emotion regulation are as follows: calculating the KL divergence between the probability distribution of the explicit emotion representation and the probability distribution of the predicted explicit emotion representation, and using the KL divergence as the intensity of emotion regulation.

10. The method according to claim 1, characterized in that, The method is trained end-to-end using a joint loss function, which includes an intrinsic state loss for supervising the learning of the intrinsic emotional state, an explicit representation loss for supervising the learning of the explicit emotional representation, and a bridging prediction loss for supervising the learning of the predicted explicit emotional representation.

Images