A multi-modal affective continuous recognition method for medical treatment

Abstract

The application discloses a multi-modal emotion continuous recognition method for medical treatment, and belongs to the field of artificial intelligence and medical information technology. A neural ordinary differential equation (Neural ODEs) framework is established to simulate the dynamic change process of emotion and overcome the static nature and discreteness of emotion modeling in traditional methods. Through causal inference technology, the physiological differences unrelated to individual emotion characteristics are separated, and the cross-individual generalization ability is improved. By using a self-supervised learning method, the synergistic effect between EEG and eye movement signals is improved through cross-modal contrast learning, and the emotion recognition accuracy is enhanced. Through a dynamic sparse attention mechanism, the computational complexity is reduced, and the key time segments in emotion recognition are focused on. Through multi-task joint learning, the model learns multiple tasks such as emotion intensity regression and subject identity recognition while performing emotion classification, thereby improving the individualized emotion recognition capability.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and medical information technology, and in particular to a multi-modal emotion continuous recognition method for medical treatment. BACKGROUND

[0002] Emotion recognition has broad application prospects in human-computer interaction, mental health assessment, and neurological disease diagnosis. Traditional single-modal emotion recognition methods (such as based on facial expressions or speech) are susceptible to environmental, disguise, and cultural differences, and have limited recognition accuracy.

[0003] In recent years, multi-modal emotion recognition, by fusing information from different signal modalities (such as electroencephalogram (EEG), eye movement signals, and electrocardiogram signals), can effectively improve the accuracy and robustness of emotion recognition. In particular, physiological signals such as EEG (electroencephalogram) and eye movement signals have unique advantages in emotion recognition. EEG signals can reflect changes in brain neural activity, while eye movement signals reveal the visual attention patterns of individuals in emotional states. Despite this, current multi-modal emotion recognition methods still face the following challenges:

[0004] Discretization of emotion modeling: Existing methods usually treat emotions as static discrete states and recognize them through static classifiers, ignoring the continuous change process of emotions. Emotions are essentially a continuous, dynamic physiological and psychological process. This discretization and static modeling approach cannot accurately capture the subtle changes and transitional states of emotions, resulting in rough and unnatural recognition results.

[0005] Poor cross-subject generalization: Existing methods have poor adaptability between different subjects and are easily affected by individual physiological differences. Specifically, physiological signals such as EEG and eye movement have significant individual differences, which are caused by factors such as electrode impedance, scalp structure, and cognitive habits. Existing models are prone to overfitting to the specific patterns of the training individual, and their recognition performance drops sharply when faced with new subjects, severely limiting their universality in clinical practice.

[0006] Poor signal fusion effect: Although multi-modal fusion has been proven to be effective, how to deeply explore the internal correlation and complementarity between different modalities (such as EEG and eye movement) remains a challenge. Multi-modal signal fusion methods have not fully exploited the complementarity of EEG signals and eye movement signals, failing to improve the accuracy of multi-modal emotion recognition. Simple feature concatenation or early fusion strategies cannot handle the asymmetry and asynchrony between modalities, failing to fully leverage the advantages of multi-modal data.

[0007] High computational complexity: Deep learning-based models have high computational overhead when processing long time series data, making real-time applications difficult to implement and unable to meet the demand for real-time emotion monitoring and analysis in medical scenarios.

[0008] Therefore, there is an urgent need for a new type of emotion recognition method that can continuously and dynamically model emotions, effectively overcome individual differences, deeply integrate multi-modal information, and has high computational efficiency, in order to promote the practical application of affective computing in precision medicine and remote health monitoring. SUMMARY

[0009] The present application aims to overcome the problems existing in the prior art and provides a multi-modal continuous emotion recognition method for medical treatment.

[0010] The purpose of the present application is achieved by the following technical solutions:

[0011] A multi-modal continuous emotion recognition method for medical treatment, comprising the following steps:

[0012] S1. Collecting multi-modal physiological signal data of the subjects, including at least electroencephalogram (EEG) signals and eye movement signals, and preprocessing;

[0013] S2. Receiving the preprocessed multi-modal physiological signals and constructing a dynamic emotion recognition model based on neural differential equations;

[0014] S3. Training the dynamic emotion recognition model, specifically including:

[0015] Causally intervening in the input or intermediate features of the dynamic emotion recognition model, generating adversarial samples using counterfactual reasoning, and comparing the similarities between electroencephalogram (EEG) signals and eye movement signals through cross-modal comparison learning;

[0016] Selecting the most relevant nodes for calculation through a dynamic sparse attention mechanism;

[0017] Optimizing the loss functions of the emotion classification task, the emotion intensity regression task, the individual identity adversarial task, and the contrastive learning task jointly;

[0018] S4. Using the trained dynamic emotion recognition model to perform forward propagation on the input unknown multi-modal physiological signals, and outputting continuous emotion state category and intensity prediction results.

[0019] In some embodiments, the dynamic emotion recognition model based on neural differential equations is constructed, including:

[0020] The evolution of the emotional state is defined by the following differential equation:

[0021]

[0022] where, is the hidden state representation of emotion, representing the state of emotion at time t; is the input physiological signal; θ is a nonlinear dynamic function learned by a neural network, describing the change of emotional state over time; θ is a trainable parameter. These are trainable weights.

[0023] In some embodiments, the time step in the dynamic emotion recognition model is adjusted according to the rate of emotion change using an adaptive time step strategy.

[0024] In some embodiments, the causal intervention is based on the Pearl causal model.

[0025] In some embodiments, the method of learning the similarity between EEG signals and eye-tracking signals through cross-modal comparison includes:

[0026] EEG and eye-tracking signals within the same time window are considered positive samples, while signals from different time windows are considered negative samples. The dynamic emotion recognition model is trained by minimizing the similarity between negative samples and maximizing the similarity between positive samples.

[0027] In some embodiments, the step of selecting the most relevant node for computation using a dynamic sparse attention mechanism includes:

[0028] The Top-K related nodes at each time step are dynamically selected through a gating mechanism.

[0029] In some embodiments, the total loss function of the dynamic emotion recognition model is:

[0030]

[0031] in, Indicates the total loss. Indicates the loss in the sentiment classification task. This indicates that the intensity of emotion regresses to the task loss. This indicates individual identity in the face of mission losses. Indicates the learning task loss in comparison. This represents the decoupling loss function. , , , This represents the balancing weight of each loss term.

[0032] It should be further noted that the technical features corresponding to the above embodiments can be combined or substituted with each other to form new technical solutions without conflict.

[0033] Compared with the prior art, the beneficial effects of the present invention are:

[0034] 1. This invention addresses the problem of static emotion modeling in traditional emotion recognition methods by introducing a Neural Differential Equations (ODEs) framework. By modeling EEG and eye-tracking signals as continuous-time dynamic systems, the Neural ODEs framework can accurately simulate the dynamic evolution of emotions, avoiding the shortcomings of static modeling. Using an ODE solver, the model can adapt to the time step, capturing the changes in emotional state over time, thereby improving the accuracy of emotion recognition. This improvement not only enhances the accuracy of emotion modeling but also makes emotion recognition more consistent with the natural evolution of emotions.

[0035] 2. This invention employs causal inference technology, successfully addressing the problem of poor generalization ability in existing emotion recognition models. By introducing counterfactual reasoning, emotional characteristics are separated from individual-independent physiological differences, enabling the model to operate stably on new subject data. Through this technique, the model can not only recognize the same emotional state but also eliminate irrelevant differences between individuals, significantly improving cross-individual adaptability and solving the problem of decreased recognition accuracy caused by individual differences.

[0036] 3. In terms of computational efficiency, this invention reduces computational complexity by dynamically selecting the Top-K most relevant nodes at each time step for calculation. Reduce to This significantly reduces computational overhead while improving inference speed. The introduction of this technology not only enables the model to respond in real time but also allows it to process longer sequence data, thus achieving real-time emotion recognition. Attached Figure Description

[0037] Figure 1 This is a flowchart of a multimodal emotion continuous recognition method for medical applications according to the present invention;

[0038] Figure 2 This is a schematic diagram of the neural differential equations (Neural ODEs) framework of this invention. Detailed Implementation

[0039] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] It should be noted that the defects in the solutions in the prior art are all the results of the inventors' practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the embodiments of this application in the following text should be the inventors' contributions to this application in the process of invention and creation, and should not be understood as technical content known to those skilled in the art.

[0041] Reference Figure 1 In one exemplary embodiment, a method for multimodal emotion continuity recognition in medical settings includes the following steps:

[0042] S1. Collect multimodal physiological signal data from the subjects, including at least EEG signals and eye movement signals, and perform preprocessing;

[0043] S2. Receive preprocessed multimodal physiological signals and construct a dynamic emotion recognition model based on neural differential equations;

[0044] S3. Training the dynamic emotion recognition model, specifically including:

[0045] Causal intervention is applied to the input or intermediate features of the dynamic emotion recognition model, and adversarial examples are generated using counterfactual reasoning; the similarity between EEG signals and eye movement signals is learned through cross-modal comparative learning.

[0046] The most relevant node is selected for computation using a dynamic sparse attention mechanism;

[0047] By jointly optimizing the loss functions for emotion classification tasks, emotion intensity regression tasks, individual identity adversarial tasks, and contrastive learning tasks;

[0048] S4. The trained dynamic emotion recognition model is used to propagate the unknown multimodal physiological signals into the forward direction, and the continuous emotion state category and intensity prediction results are output.

[0049] Emotional states are a dynamic evolutionary process, while traditional methods often treat emotions as discrete, static categories. To overcome this problem, this invention introduces Neural Differential Equations (Neural ODEs) to simulate the evolution of emotional states over time. The framework of Neural Differential Equations is as follows: Figure 2 As shown, through neural ODEs, the model can perform more continuous and detailed modeling of emotional states, enabling accurate prediction of emotions at every moment.

[0050] For example, in step S2, constructing a dynamic emotion recognition model based on neural differential equations includes:

[0051] The evolution of emotional states is defined by the following differential equation:

[0052]

[0053] in, Let be the latent state representation of emotion, representing the state of emotion at time t; For inputting physiological signals (such as EEG or eye movement signals); θ is a nonlinear dynamic function learned by a neural network, describing the change of emotional state over time; θ is a trainable parameter. For trainable weights, the linear driving term, nonlinear enhancement term, and periodic perturbation term are adjusted respectively.

[0054] Furthermore, in the dynamic emotion recognition model, the time step is adjusted according to the rate of emotion change using an adaptive time step strategy. Since emotions may change at different rates, neural ODEs allow the model to adaptively adjust the time step based on the rate of emotion change. Rapidly changing emotional states (such as fear and anger) are modeled using smaller time steps for finer detail, while more stable emotional states (such as relaxation and happiness) are modeled using larger time steps. Through this adaptive time step mechanism, the model can more accurately capture subtle fluctuations in emotional states. The adaptive time step strategy is as follows:

[0055]

[0056] in, For adaptive time step; Let L2 norm represent the magnitude of the state change; η is the step size scaling factor. δ is a regularization parameter that controls the sensitivity of the step size to the rate of change; δ is a minimal constant to avoid division by zero.

[0057] Furthermore, to solve the neural differential equations, this invention uses a numerical solver and updates the parameters of the neural network through a backpropagation algorithm. Using gradient descent, the model gradually approaches the optimal parameters during training to minimize the error between the predicted sentiment state and the actual sentiment label.

[0058] The gradient is calculated in backpropagation as follows:

[0059]

[0060] Where L is the loss function and T is the total duration of the time series. θ is the instantaneous loss at time t, and θ is the parameter of the ODE function. It is the Jacobian matrix of the hidden state with respect to the parameters.

[0061] Traditional emotion recognition models typically rely on individual physiological differences (such as individual differences in EEG and eye-tracking signals). These differences lead to low generalization ability when processing data from different individuals. Therefore, this invention introduces a causal intervention module, combined with counterfactual reasoning techniques, to address the generalization problem of cross-individual emotion recognition at its root.

[0062] For example, the causal intervention first constructs a causal graph that describes the causal relationship between emotional traits and confounding factors (such as electrode impedance and individual differences). The model adjusts the input signal through the causal intervention to effectively separate emotional traits from confounding factors.

[0063] Construct a causal graph G, with nodes including input signal X, emotional state Y, and individual variable Z. Based on the Pearl causal model, the intervention operation can be formalized as follows:

[0064]

[0065] in, Causal intervention (forced input signal) ). The emotional state is represented by a specific value y. Confounding variables (individual differences / electrode impedance, etc.) take a specific value z. This represents the prior distribution of the confounding variable. This formula removes the confounding effects from Z in the X→Y path.

[0066] To enhance the robustness of cross-individual emotion recognition, this invention employs counterfactual reasoning to generate adversarial examples. Counterfactual reasoning simulates changes in emotional characteristics among different individuals, helping the model learn emotional characteristics unaffected by individual differences. Through this technique, the model can maintain high recognition efficiency even when faced with data from diverse individuals.

[0067] The formula for counterfactual reasoning is as follows:

[0068]

[0069] in, It is a sentiment prediction result generated by counterfactual reasoning. It is the input signal after causal intervention, which simulates the changes in emotional characteristics between different individuals. Causal_Graph is a causal graph that represents the causal relationship between emotional state and other confounding factors (such as individual differences).

[0070] Adversarial examples generated through causal intervention help the model learn cross-individual generalization ability, enabling the model to maintain good performance on data with unknown subjects.

[0071] For example, dynamic emotion recognition models employ multimodal self-supervised pre-training. However, the annotation of physiological signals often requires professional personnel, which is costly and results in limited data. To address this issue, this invention employs cross-modal self-supervised learning. Through cross-modal comparative learning, the model can effectively learn the relationship between EEG signals and eye-tracking signals even without a large amount of labeled data, thereby improving the performance of emotion recognition.

[0072] Specifically, the design goal of the cross-modal contrastive learning task is to maximize the similarity between EEG signals and eye-tracking signals. In particular, EEG and eye-tracking signals within the same time window are considered positive samples, while signals from different time windows are considered negative samples. The model is trained by minimizing the similarity between negative samples and maximizing the similarity between positive samples.

[0073] The loss function for contrastive learning is as follows:

[0074]

[0075] in, and These are the embedding vectors of EEG signals and eye-tracking signals within the same time window, both of which are positive samples; Cosine similarity; Here, N represents the temperature parameter; N is the number of negative samples. Negative samples represent EEG / eye movement signals from different time windows.

[0076] For example, when processing long sequences of physiological signals, the full attention mechanism of the Transformer is computationally intensive, especially when the sequence length is long, with a computational complexity of O(n). This limits the model's processing power. Specifically, for each input time step, the model needs to calculate its attention weights with all other time steps. The calculation formula is as follows:

[0077]

[0078] Where Q is the query matrix, representing the input features at the current time step. K is the key matrix, representing the input features at other time steps. V is the value matrix, representing the representation associated with each time step. This is the dimension of the key vector. The computational complexity is O(n log n). This is because it is necessary to calculate all similarities between N time steps.

[0079] To overcome the problem of computational redundancy, this invention introduces a dynamic sparse attention mechanism, which dynamically selects the Top-K relevant nodes for computation at each time step through a gating mechanism. This reduces unnecessary computation and memory consumption.

[0080] In dynamic sparse attention mechanisms, gating selects the Top-K most relevant nodes by calculating the relevance weights at each time step. The core of this process is calculating the similarity between each node, then ranking them according to this similarity to select the most relevant node.

[0081] Calculate similarity:

[0082]

[0083] Where Q and K are the query matrix and the key matrix, respectively. It is the dimension of the key / query vector.

[0084] By sorting the calculated similarities, the Top-K relevant nodes at each time step are selected:

[0085]

[0086] in, The function returns sorted indices; It is a similarity matrix; K is the number of Top-K related nodes retained.

[0087] Furthermore, to improve the accuracy of emotion recognition, this invention introduces a multi-task joint learning framework, which combines emotion classification, emotion intensity regression, individual identity adversarial tasks, and contrastive learning tasks. Each task has a different loss function, and through joint optimization, the model can simultaneously learn emotion classification, emotion intensity, and robust cross-individual features during training.

[0088] Loss in sentiment classification task (main task):

[0089]

[0090] in, As an emotional tag, To predict probabilities.

[0091] Loss in the emotion intensity regression task (auxiliary task):

[0092]

[0093] in, To reflect the true intensity of emotion, To predict the intensity of emotion, N represents the number of samples.

[0094] The individual identity adversarial task (auxiliary task) uses cross-entropy loss:

[0095]

[0096] The individual identity adversarial task loss is used to suppress the influence of individual identity on sentiment recognition, enabling the model to recognize irrelevant sentiment features, where s represents the true individual label of the sample. For identity feature extractor (input: hidden state h, output: identity probability distribution). To predict the probability of identity s.

[0097] Contrast learning loss (auxiliary task) loss:

[0098]

[0099] The embedding representation of the sample is given, where τ is the temperature parameter. It is a positive sample. These are negative samples. Contrastive learning loss ensures that the model learns consistent and complementary modal representations (such as common features of EEG and eye-tracking signals) by maximizing the similarity of similar modalities and minimizing the distance between different modalities.

[0100] To address the interference of individual differences in emotion recognition, this invention designs a decoupled representation learning approach. By decoupling the encoder, emotion features and individual identity features are learned separately, allowing the model to focus on the essential characteristics of emotion without being influenced by individual identity.

[0101] Decoupling loss function:

[0102]

[0103] in, As an emotional characteristic, For individual identity characteristics, This represents the Frobenius norm.

[0104] Furthermore, the joint total loss is a comprehensive loss function used to optimize the model in a multi-task learning framework. It combines the objectives of multiple tasks (such as sentiment classification, regression, adversarial learning, and contrastive learning). The purpose of this loss function is to balance the influence of different tasks during training, ensuring that the model can optimize multiple task objectives simultaneously without excessively favoring any one task. The total loss function for the dynamic sentiment recognition model is:

[0105]

[0106] Each loss term corresponds to a different task objective, where... Indicates the total loss. Indicates the loss in the sentiment classification task. This indicates that the intensity of emotion regresses to the task loss. This indicates individual identity in the face of mission losses. Indicates the learning task loss in comparison. This represents the decoupling loss function. , , , This represents the balancing weight of each loss term, used to adjust the contribution of each task to the total loss. Controlling the impact of regression loss on total loss, higher Weighting increases the contribution of the regression task (emotional intensity) to the total loss. The impact of controllable losses on total losses is relatively high. Weights enhance the model's ability to eliminate individual identity features. Controlling for the impact of contrastive learning loss. Higher... Weights facilitate alignment between different modal representations. The impact of controlling mutual information decoupling loss. Higher... Weights reduce the correlation between emotional expression and individual identity, thereby improving generalization ability across individuals.

[0107] During training, these weights are typically adjusted using cross-validation or adaptive optimization strategies to ensure that the losses of each task are properly balanced in the final joint loss.

[0108] The above detailed embodiments are a description of the present invention. It should not be considered that the specific embodiments of the present invention are limited to these descriptions. For those skilled in the art, several simple deductions and substitutions can be made without departing from the concept of the present invention, and all of these should be considered to fall within the protection scope of the present invention.

Claims

1. A multimodal emotion continuity recognition method for medical applications, characterized in that, Includes the following steps: S1. Collect multimodal physiological signal data from the subjects, including at least EEG signals and eye movement signals, and perform preprocessing; S2. Receive preprocessed multimodal physiological signals and construct a dynamic emotion recognition model based on neural differential equations; The construction of the dynamic emotion recognition model based on neural differential equations includes: The evolution of emotional states is defined by the following differential equation: ,in, Let be the latent state representation of emotion, representing the state of emotion at time t; To input physiological signals; θ is a nonlinear dynamic function learned by a neural network, describing the change of emotional state over time; θ is a trainable parameter. These are trainable weights; S3. Training the dynamic emotion recognition model, specifically including: Causal intervention is applied to the input or intermediate features of the dynamic emotion recognition model, and adversarial examples are generated using counterfactual reasoning; the similarity between EEG signals and eye movement signals is learned through cross-modal comparative learning. The most relevant node is selected for computation using a dynamic sparse attention mechanism; By jointly optimizing the loss functions for emotion classification tasks, emotion intensity regression tasks, individual identity adversarial tasks, and contrastive learning tasks; S4. The trained dynamic emotion recognition model is used to propagate the unknown multimodal physiological signals into the forward direction, and the continuous emotion state category and intensity prediction results are output.

2. The multimodal emotion continuous recognition method for medical applications according to claim 1, characterized in that, In the dynamic emotion recognition model, the time step is adjusted according to the rate of emotion change using an adaptive time step strategy.

3. The multimodal emotion continuity recognition method for medical applications according to claim 1, characterized in that, The causal intervention is based on the Pearl causal model.

4. The multimodal emotion continuity recognition method for medical applications according to claim 1, characterized in that, The method of learning the similarity between EEG signals and eye movement signals through cross-modal comparison includes: EEG and eye-tracking signals within the same time window are considered positive samples, while signals from different time windows are considered negative samples. The dynamic emotion recognition model is trained by minimizing the similarity between negative samples and maximizing the similarity between positive samples.

5. The multimodal emotion continuity recognition method for medical applications according to claim 1, characterized in that, The selection of the most relevant node for computation through a dynamic sparse attention mechanism includes: The Top-K related nodes at each time step are dynamically selected through a gating mechanism.

6. The multimodal emotion continuity recognition method for medical applications according to claim 1, characterized in that, The total loss function of the dynamic emotion recognition model is: ,in, Indicates the total loss. Indicates the loss in the sentiment classification task. This indicates that the intensity of emotion regresses to the task loss. This indicates individual identity in the face of mission losses. This indicates the loss in comparison learning tasks. This represents the decoupling loss function. , , , This represents the balancing weight of each loss term.

Images