Emotional state detection method, system, device, medium and product

Abstract

The invention provides an emotional state detection method, system and device, a medium and a product, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining target multi-modal data corresponding to a to-be-detected user; and inputting the target multi-modal data into a multi-modal emotional state detection model to obtain an emotional detection result of the to-be-detected user output by the multi-modal emotional state detection model, the multi-modal emotional state detection model being based on sample multi-modal data. Training the cross-modal space-time diagram neural network to obtain the cross-modal space-time diagram neural network; the cross-modal space-time diagram neural network is trained through a differentiable gating mechanism to obtain a weight coefficient corresponding to each modal; the multi-modal emotional state detection model comprises a full-modal emotional state detection model and a privacy modal emotional state detection model. According to the invention, the real emotional state of the user can be reflected more accurately.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, system, device, medium, and product for detecting emotional states. Background Technology

[0002] As people pay increasing attention to their mental health, the demand for intelligent monitoring of their emotions and psychological states is also growing stronger. Whether in the field of healthcare, education, or the workplace, intelligent monitoring of emotions and psychological states has significant practical implications for understanding an individual's psychological state and providing targeted support and assistance.

[0003] Currently, in the field of emotion and psychological state monitoring, traditional methods mainly rely on the analysis of information from a single modality. These single-modality technologies can provide valuable information for the monitoring of emotion and psychological state to a certain extent.

[0004] However, traditional methods relying on a single modality have significant limitations, failing to comprehensively and accurately reflect an individual's true emotions and psychological state. Therefore, there is an urgent need for an emotion state detection method, system, device, medium, and product to address these issues. Summary of the Invention

[0005] In view of the problems existing in the prior art, the present invention provides an emotion state detection method, system, device, medium and product.

[0006] This invention provides a method for detecting emotional states, comprising: Obtain the target multimodal data corresponding to the user to be detected; The target multimodal data is input into a multimodal emotion state detection model to obtain the emotion detection result of the user to be detected output by the multimodal emotion state detection model. The multimodal emotion state detection model is obtained by training a cross-modal spatiotemporal graph neural network based on sample multimodal data. The cross-modal spatiotemporal graph neural network is trained to obtain the weight coefficients corresponding to each modality through a differentiable gating mechanism.

[0007] According to the present invention, an emotion state detection method is provided, wherein the multimodal emotion state detection model includes a full-modal emotion state detection model and a privacy-modal emotion state detection model; the target multimodal data includes first multimodal data and second multimodal data; The step of inputting the target multimodal data into a multimodal emotion state detection model to obtain the emotion detection result of the user to be detected output by the multimodal emotion state detection model includes: The first multimodal data is input into the full-modal emotion state detection model to obtain the emotion detection result of the user to be detected output by the full-modal emotion state detection model. The first multimodal data is constructed based on the target speech feature signal, target vital sign signal and target video feature information of the user to be detected. The target vital sign signal is obtained based on millimeter-wave radar. Alternatively, the second multimodal data can be input into the privacy modality emotion state detection model to obtain the emotion detection result of the user to be detected output by the privacy modality emotion state detection model, wherein the second multimodal data is constructed based on the target speech feature signal and the target vital sign signal of the user to be detected.

[0008] According to an emotion state detection method provided by the present invention, before inputting the target multimodal data into a multimodal emotion state detection model to obtain the emotion detection result of the user to be detected output by the multimodal emotion state detection model, the method further includes: Based on the short-time Fourier transform and Mel frequency cepstral coefficients, the target time-frequency features corresponding to the target speech feature signal are obtained; The target time-frequency features are input into the speech feature extraction model to obtain the target emotion category features output by the speech feature extraction model. The speech feature extraction model is constructed based on a convolutional neural network and a bidirectional long short-term memory network. Based on Kalman filtering, extended Kalman filtering, or variational Bayesian filtering, the target vital sign signal is denoised to obtain the target physiological filtered features. Based on convolutional neural networks and self-attention mechanisms, feature extraction is performed on the target video feature information to obtain target facial expression features and target body posture features; The first multimodal data is constructed based on the target emotion category features, the target physiological filtering features, the target facial expression features, and the target body posture features. The second multimodal data is constructed based on the target emotion category features and the target physiological filtering features.

[0009] According to the present invention, an emotion state detection method is provided, wherein the full-modal emotion state detection model is trained through the following steps: Based on the sample emotion category features, sample physiological filtering features, sample facial expression features, and sample body posture features, sample multimodal data corresponding to each modality is constructed. The sample multimodal data is a set of three nodes, including temporal frame, spatial location, and modal features. The various sample multimodal data are connected by hyperedges. The sample multimodal data is labeled with corresponding emotion state tags to obtain the training sample set; Based on the training sample set, the cross-modal spatiotemporal graph neural network is trained to obtain the full-modal emotion state detection model.

[0010] According to the present invention, an emotion state detection method is provided, wherein the privacy modality emotion state detection model is obtained through the following steps: The full-modal emotion state detection model is used as the teacher model; Based on knowledge distillation, the attention map and hidden layer output features in the teacher model are transferred to the student network to obtain the privacy modality emotion state detection model. The student network is obtained by training the cross-modal spatiotemporal graph neural network based on the sample time-frequency features and the sample physiological filtering features.

[0011] According to the present invention, an emotion state detection method is provided, wherein training the cross-modal spatiotemporal graph neural network based on the training sample set to obtain the full-modal emotion state detection model includes: Based on the differentiable gating mechanism, the weight matrix in the dynamic graph convolution operator of the cross-modal spatiotemporal graph neural network is adjusted. After determining that the training results meet the preset conditions, the full-modal emotion state detection model is obtained. The dynamic graph convolution operator is: ; in, and These are the nodes corresponding to different samples of multimodal data. For nodes In the cross-modal spatiotemporal graph neural network, the first Layer feature representation, It is a non-linear activation function. Represents nodes The set of all connected superedges. For super-edge The set of all connected nodes. Indicates the superedge Number of connected nodes; This indicates the first time after adjustment by the differentiable gating mechanism. The first layer The weight matrix corresponding to each mode; For nodes In the The feature vector representation of the layer, For nodes With nodes The time distance between them This indicates a feature splicing operation.

[0012] The present invention also provides an emotion state detection system, comprising: The multimodal data acquisition module is used to acquire the target multimodal data corresponding to the user to be detected; The emotion state detection module is used to input the target multimodal data into the multimodal emotion state detection model to obtain the emotion detection result of the user to be detected output by the multimodal emotion state detection model. The multimodal emotion state detection model is obtained by training a cross-modal spatiotemporal graph neural network based on sample multimodal data. The cross-modal spatiotemporal graph neural network is trained to obtain the weight coefficients corresponding to each modality through a differentiable gating mechanism.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the emotional state detection method as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the emotion state detection method as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the emotional state detection method as described above.

[0016] This invention specifically includes the following core innovations: I. Distinguishing between full-modal and privacy-modal modes: For the first time, a dual-path architecture of privacy-modal (voice + millimeter-wave radar) and full-modal (video + voice + millimeter-wave radar) is proposed to meet the needs of emotion assessment in privacy scenarios, while providing high-precision full-modal analysis with video support; II. Cross-modal spatiotemporal graph neural network: Unifies the temporal, spatial and modal dimensions in a third-order hypergraph, and uses a differentiable gating mechanism to model the cross-spatiotemporal associations of different modalities, effectively improving the completeness and accuracy of emotional semantic representation; 3. Progressive feature distillation: Attention distillation is performed from the full-modal teacher network to the privacy-modal student network to effectively transfer model knowledge containing video information to the video-free path, ensuring that the privacy modality can still maintain high recognition accuracy. IV. Adaptive Physiological Signal Filtering: Introducing variational Bayesian filtering or extended Kalman filtering, high-precision estimation of millimeter-wave radar vital signs signals under motion and interference scenarios is achieved, providing more stable and reliable physiological feature support for the assessment of emotions and psychological states.

[0017] The emotion state detection method, system, device, medium, and product provided by this invention acquires target multimodal data of the user to be detected. This data covers information from different modalities and together constitutes the data foundation for emotion state analysis. Then, the acquired target multimodal data is input into a multimodal emotion state detection model, which outputs the emotion detection result of the user to be detected. Since the emotion detection result integrates the information in the multimodal data and considers the spatiotemporal dynamic characteristics and importance weights between different modal data, it can more accurately reflect the user's true emotional state. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating the emotional state detection method provided by this invention; Figure 2 This is a schematic diagram of the structure of the emotion state detection system provided by the present invention; Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] Currently, in the field of emotion and psychological state monitoring, traditional methods mainly rely on single-modal information analysis. Among these, speech analysis technology extracts and analyzes an individual's speech features, such as pitch, speech rate, volume, and pauses, to infer their emotional state. For example, when a person is angry, their pitch may rise and their speech rate may increase. Facial expression recognition technology uses computer vision algorithms to capture and identify an individual's facial muscle movements and micro-expressions to determine their emotions. Common facial expressions include happiness, sadness, surprise, and anger, each with its specific facial features. Physiological signal monitoring technology focuses on an individual's physiological indicators, such as heart rate, blood pressure, and skin conductance. Changes in these physiological signals can reflect an individual's stress level and emotional state. For example, when a person feels nervous, their heart rate may increase, and their skin conductance may strengthen. These single-modal technologies can provide valuable information for monitoring emotions and psychological states to a certain extent.

[0022] However, traditional methods relying on a single modality have significant limitations, failing to comprehensively and accurately reflect an individual's true emotions and psychological state. On one hand, emotions and psychological states are complex and multi-dimensional; single-modal information often only captures one aspect of their characteristics, neglecting other important cues. For example, a person may appear calm verbally, but their facial expressions reveal inner anxiety, or abnormal fluctuations in physiological signals may occur. Relying solely on speech analysis or facial expression recognition cannot fully represent their true emotional state. On the other hand, individuals may express emotions differently in different situations. Single-modal methods are easily affected by external factors, leading to inaccurate monitoring results. For instance, in social situations, people may deliberately conceal their true emotions, making facial expressions or speech features unable to accurately reflect their inner psychological state.

[0023] Furthermore, information from different modalities is interconnected and complementary. Most existing multimodal fusion schemes simply cascade data at the feature or decision layers, ignoring the temporal and spatial relationships between different modalities. Simultaneously, they fail to differentiate between modalities with varying privacy sensitivities, potentially leading to serious privacy and compliance issues if data is misused. Moreover, monitoring emotions and psychological states requires strong dynamic adaptability to handle different scenarios and changes. For example, in motion or interference-prone environments, an individual's emotional expression and physiological signals may change. However, existing methods cannot adjust to real-time conditions, resulting in decreased accuracy of Kalman filtering in motion or interference scenarios, failing to meet the high requirements for real-time performance and accuracy.

[0024] To address the problems existing in the prior art, this invention provides a method for detecting individual emotions and psychological states based on full-modal perception and extended-modal spatiotemporal graph neural networks. By integrating voice, millimeter-wave radar data, and video data, it balances privacy protection with high-precision dynamic perception, achieving more comprehensive and accurate monitoring of emotions and psychological states.

[0025] Figure 1 This is a flowchart illustrating the emotion state detection method provided by the present invention, as shown below. Figure 1 As shown, the present invention provides a method for detecting emotional states, including: Step 101: Obtain the target multimodal data corresponding to the user to be detected.

[0026] In this invention, multimodal data refers to data from different perceptual modalities or information sources. In the scenario of emotion state detection, common modalities include, but are not limited to, the following: Speech modality: includes speech features such as the content of a user's speech, tone, speed, volume, and pauses. For example, a person may raise their volume and speed up their speech when excited, while their tone may be low and their speech may be slow when frustrated.

[0027] Facial expression modality: It reflects emotions by capturing the user's facial muscle movements and micro-expressions. For example, raising or frowning eyebrows, widening or squinting eyes, and raising or lowering the corners of the mouth can all convey different emotional information, such as raising the corners of the mouth when happy and frowning when angry.

[0028] Physiological signal modalities: These involve physiological indicators such as the user's heart rate, blood pressure, and respiratory rate. Changes in these physiological signals are closely related to emotional state; for example, when a person feels nervous, their heart rate may increase. Preferably, the physiological signal modalities in this invention can be acquired using non-contact millimeter-wave radar.

[0029] Body language modality: A user's body posture, gestures, and gait. For example, crossing one's arms may indicate defensiveness or anxiety, while standing tall and proud may convey confidence and positive emotions.

[0030] In this invention, the user to be detected refers to a specific individual who needs to have their emotional state detected. These individuals can be anyone who has an emotional monitoring need, such as a patient undergoing psychological evaluation in a medical setting, a student who needs to understand their learning status in an educational setting, or an employee who needs to pay attention to work pressure in a workplace environment.

[0031] Furthermore, this invention utilizes various specialized sensors to collect data from different modalities. For example, it uses microphones to collect voice data, cameras to capture facial expressions and body movements, and wearable devices (such as smart bracelets and electrocardiogram monitors) to acquire physiological signal data. These sensors can be integrated into a single device, such as smart glasses or smart clothing, for convenient wear and use by users, enabling simultaneous acquisition of multimodal data.

[0032] In one embodiment, the user to be tested can actively upload their multimodal data through an application or platform. For example, the user can use a mobile phone to record their own voice, take videos of facial expressions, and upload relevant physiological data (such as data exported from health monitoring devices) for emotional state detection. If the relevant data of the user to be tested is already stored in other systems or platforms, this data can be obtained through data interfaces. For example, the information system of a medical institution may store some of the patient's physiological data and medical records. By interfacing with the interface of this system, relevant data for emotional state detection can be obtained.

[0033] In this invention, data from different modalities contains rich information about the user's emotional state. This information complements and corroborates each other, enabling a more comprehensive capture of subtle changes in the user's emotional state and improving the reliability and effectiveness of detection. For example, when detecting whether a user is in an anxious state, voice analysis alone may not be accurate, as the user may consciously control their vocal performance. However, by combining multimodal data such as facial expressions, physiological signals, and body movements, a comprehensive judgment can be made from multiple perspectives, more accurately identifying the user's anxiety.

[0034] Step 102: Input the target multimodal data into the multimodal emotion state detection model to obtain the emotion detection result of the user to be detected output by the multimodal emotion state detection model. The multimodal emotion state detection model is obtained by training a cross-modal spatiotemporal graph neural network based on sample multimodal data. The cross-modal spatiotemporal graph neural network is trained to obtain the weight coefficients corresponding to each modality through a differentiable gating mechanism.

[0035] In this invention, the acquired target multimodal data of the user to be detected is input into a multimodal emotion state detection model for emotion state detection. After a series of complex calculations and analyses, the multimodal emotion state detection model outputs the emotion detection result of the user to be detected. In this invention, the multimodal emotion state detection model is obtained by training a cross-modal spatiotemporal graph neural network (DST-GNN) based on sample multimodal data. Furthermore, a differentiable gating mechanism is used to determine the weight coefficients corresponding to each modality during the training of the cross-modal spatiotemporal graph neural network.

[0036] Specifically, multimodal sample data refers to data collected from a large number of different individuals and scenarios, containing multiple modalities (such as speech, facial expressions, and physiological signals), and these data all have corresponding accurate emotional state annotations. For example, a set of data containing speech, facial expressions, and physiological signals is collected, and the corresponding emotions are labeled as "happy," "sad," and "angry." By learning from this multimodal sample data, cross-modal spatiotemporal graph neural networks can grasp the complex relationship between different modal data and emotional states, thereby accurately predicting the emotional state of a new user when faced with multimodal data.

[0037] A cross-modal spatiotemporal graph neural network (CSP) is a neural network model used to process graph-structured data with spatiotemporal dynamics. In multimodal emotion detection, data from different modalities can be viewed as nodes in a graph structure, and the spatiotemporal relationships between them can be represented as edges. For example, the speech modality and the facial expression modality may have a synchronous relationship in time, which can be represented by edges in the graph. In this invention, the CSP can effectively capture the dynamic changes of multimodal data in time and space. For example, in a video, a user's facial expressions and speech change over time. The CSP can analyze the relationships between these changes, thereby better understanding the user's emotional state.

[0038] Furthermore, differentiable gating is a method for automatically learning the importance weights of data from different modalities. When training a cross-modal spatiotemporal graphical neural network, the differentiable gating mechanism dynamically adjusts the weight coefficients corresponding to each modality based on the characteristics of the multimodal data samples. These weight coefficients reflect the contribution of different modalities to emotion state detection. For example, in some cases, the speech modality may be more crucial for judging emotional states, so the differentiable gating mechanism will assign a higher weight to the speech modality; while in other cases, the facial expression modality may contain more emotional information, and its weight will be increased accordingly. In this way, the cross-modal spatiotemporal graphical neural network can adaptively fuse data from different modalities, improving the accuracy of emotion state detection.

[0039] Finally, after processing by the multimodal emotion detection model, the final output is the emotion detection result for the user being tested. This result can be a specific emotion category, such as "happy," "sad," "angry," or "anxious," or it can be a numerical value or vector representing the intensity of the emotion. This result can reflect the emotional state of the user at a specific moment or in a specific scenario, providing important information for applications in related fields. For example, in the medical field, doctors can understand the patient's psychological state based on the emotion detection results and formulate more appropriate treatment plans; in the education field, teachers can adjust their teaching methods based on students' emotional states to improve teaching effectiveness.

[0040] The emotion state detection method provided by this invention acquires target multimodal data of the user to be detected. This data covers information from different modalities and together constitutes the data foundation for emotion state analysis. Then, the acquired target multimodal data is input into a multimodal emotion state detection model, which outputs the emotion detection result of the user to be detected. Since the emotion detection result integrates the information in the multimodal data and considers the spatiotemporal dynamic characteristics and importance weights between different modal data, it can more accurately reflect the user's true emotional state.

[0041] Based on the above embodiments, the multimodal emotion state detection model includes a full-modal emotion state detection model and a privacy-modal emotion state detection model; the target multimodal data includes first multimodal data and second multimodal data; The step of inputting the target multimodal data into a multimodal emotion state detection model to obtain the emotion detection result of the user to be detected output by the multimodal emotion state detection model includes: The first multimodal data is input into the full-modal emotion state detection model to obtain the emotion detection result of the user to be detected output by the full-modal emotion state detection model. The first multimodal data is constructed based on the target speech feature signal, target vital sign signal and target video feature information of the user to be detected. The target vital sign signal is obtained based on millimeter-wave radar. Alternatively, the second multimodal data can be input into the privacy modality emotion state detection model to obtain the emotion detection result of the user to be detected output by the privacy modality emotion state detection model, wherein the second multimodal data is constructed based on the target speech feature signal and the target vital sign signal of the user to be detected.

[0042] In this invention, the multimodal emotion state detection model comprises two sub-models: a full-modal emotion state detection model and a privacy-modal emotion state detection model. These two models are designed for different data types and application scenarios to meet different needs. The target multimodal data is divided into first-modal data and second-modal data, a division based on the different components of the data and the different application scenarios.

[0043] In this invention, the first multimodal data is input into the full-modal emotion state detection model. The first multimodal data is constructed based on the target speech feature signal, target vital sign signal and target video feature information of the user to be detected. The target vital sign signal is obtained by millimeter-wave radar.

[0044] The target speech feature signal contains various features of the user's speech, such as pitch, speech rate, volume and pauses. These features can reflect the user's emotional state when speaking. For example, a high pitch may indicate excitement or agitation, while a low pitch may suggest sadness or frustration.

[0045] Target vital signs (acquired by millimeter-wave radar) are obtained non-contactly from the user's physiological vital signs, such as respiratory rate and heart rate, using millimeter-wave radar. These vital signs are closely related to the user's emotional state; when a person is in a state of tension or anxiety, their breathing and heart rate may increase.

[0046] Target video feature information involves extracting features such as the user's facial expressions and body movements from the video. Facial expressions are an important way to express emotions; different expressions correspond to different emotions. For example, a smile usually indicates happiness, while a frown may indicate dissatisfaction or confusion. Body movements can also convey emotional information; for example, crossing one's arms may indicate defensiveness or unease.

[0047] In this invention, the full-modal emotion state detection model processes and analyzes the input first multimodal data, and then outputs the emotion detection result of the user to be detected. This emotion detection result can be a specific emotion category, such as "happy," "sad," or "angry," or it can be a numerical value or vector representing the intensity of the emotion. Because the full-modal emotion state detection model comprehensively considers information from multiple modalities such as speech, physical signs, and video, its output emotion detection result is relatively more comprehensive and accurate.

[0048] In this invention, the second multimodal data can also be input into the privacy-preserving emotional state detection model. The second multimodal data is constructed based on the target speech feature signal and target vital sign signal of the user to be detected. Compared with the first multimodal data, the second multimodal data does not contain target video feature information, thus placing greater emphasis on protecting the user's privacy.

[0049] Furthermore, the privacy-preserving modality emotion detection model processes the input second multimodal data and outputs the emotion detection result for the user being tested. This model uses only data from two modalities: speech and body signs. While still reflecting the user's emotional state to a certain extent, it avoids the leakage of video data and protects the user's privacy.

[0050] Based on the above embodiments, before inputting the target multimodal data into the multimodal emotion state detection model to obtain the emotion detection result of the user to be detected output by the multimodal emotion state detection model, the method further includes: Based on the short-time Fourier transform and Mel frequency cepstral coefficients, the target time-frequency features corresponding to the target speech feature signal are obtained; The target time-frequency features are input into the speech feature extraction model to obtain the target emotion category features output by the speech feature extraction model. The speech feature extraction model is constructed based on a convolutional neural network and a bidirectional long short-term memory network. Based on Kalman filtering, extended Kalman filtering, or variational Bayesian filtering, the target vital sign signal is denoised to obtain the target physiological filtered features. Based on convolutional neural networks and self-attention mechanisms, feature extraction is performed on the target video feature information to obtain target facial expression features and target body posture features; The first multimodal data is constructed based on the target emotion category features, the target physiological filtering features, the target facial expression features, and the target body posture features. The second multimodal data is constructed based on the target emotion category features and the target physiological filtering features.

[0051] Speech signals are continuously changing in the time domain. To analyze their frequency characteristics, it is necessary to convert the time-domain signal to the frequency domain. This invention employs the Short-Time Fourier Transform (STFT) to divide the 16kHz speech signal acquired by a microphone array or speech sampling hardware into multiple short time segments. A Fourier transform is then performed on each short time segment to obtain the frequency distribution of the signal across different time periods. For example, for a dialogue, STFT can clearly show the changes in the frequency components of the speech signal at different times.

[0052] Mel-frequency cepstral coefficients (MFCCs) are characteristic parameters calculated based on the Mel-frequency scale, which can better simulate the auditory characteristics of the human ear. This invention performs a series of processes on the frequency domain signal obtained through STFT, including Mel-frequency filtering, logarithmic operations, and Discrete Cosine Transform (DCT), ultimately obtaining MFCC features, which are the target time-frequency features corresponding to the target speech feature signal. The target time-frequency features reflect the energy distribution of the speech signal on the Mel-frequency scale, capturing the time-frequency characteristics of speech. For example, different speech phonemes will exhibit different representations in the MFCC features, thus providing important evidence for subsequent emotion analysis.

[0053] Furthermore, the target time-frequency features are input into the speech feature extraction model to obtain the target emotion category features output by the speech feature extraction model. In this invention, the speech feature extraction model is constructed based on a Convolutional Neural Network (CNN) and a Bidirectional Long Short-Term Memory (Bi-LSTM) network. CNN has powerful local feature extraction capabilities and can learn local patterns in the MFCC feature matrix, thereby extracting local temporal features. For example, certain frequency combinations may be associated with specific emotions. Bi-LSTM excels at processing sequential data and can capture long-term dependencies in speech signals. Considering that emotional expression in speech may span multiple time periods, this invention uses Bi-LSTM or Transformer structures to process global temporal information.

[0054] This invention inputs the target time-frequency features into a speech feature extraction model, which then processes them through multiple layers of CNN and Bi-LSTM to output target emotion category features. The target emotion category features are an abstract representation of emotional information in the speech signal, containing key features related to different emotion categories. Ultimately, these features can be used for subsequent emotion classification tasks. In one embodiment, speech energy, pitch, and speech rate can also be extracted from the target speech feature signal as auxiliary features, combined with the target time-frequency features, and input into the speech feature extraction model to obtain more accurate emotion category features.

[0055] In this invention, the MFCC extraction process specifically involves setting the sampling rate. f s =16000Hz, frame length N =512 samples, frame shift S =256 samples, Hamming window function, extract the first 13 coefficients to obtain the audio time-frequency features, the formula is: ; in, x ( t ) is the first t Frame speech feature signal, MelSpectrum ( x ( t )) represents the Mel frequency spectrum, and DCT represents the discrete cosine transform.

[0056] In this invention, the specific architecture of the speech feature extraction model is as follows: Input layer: MFCC feature matrix, size 13×T, where T is the number of frames; CNN layer: Convolutional kernel size is 5×5, stride is 1, used to extract local features; LSTM layer: Uses bidirectional LSTM with an output dimension of 256 to capture global timing information; Output layer: Uses a Softmax classifier to predict emotion labels, such as happiness, sadness, anger, etc.

[0057] Furthermore, based on Kalman filtering, extended Kalman filtering, or variational Bayesian filtering, the target vital signs signal is denoised to obtain the target's physiological filtered features. In this invention, vital signs information such as heart rate (HR) and respiratory rate (RR) are acquired using non-contact millimeter-wave radar.

[0058] Since target vital signs signals (such as heart rate and respiratory rate) acquired by millimeter-wave radar may be affected by various noises, this invention uses Kalman filtering (KF) for preliminary noise reduction, filtering out high-frequency interference to achieve a more accurate estimation of nonlinear physiological signals.

[0059] Preferably, in this invention, an extended Kalman filter (EKF) or a variational Bayesian filter (VBF) can be used to denoise the target physiological signal. Specifically, taking the extended Kalman filter as an example, when the physiological signal exhibits nonlinear characteristics, the extended Kalman filter expands its application range by linearizing the nonlinear function, enabling more accurate modeling and estimation of nonlinear physiological signals and improving the denoising effect. In this invention, based on the extended Kalman filter, the modeling of the nonlinear physiological signal is updated as follows: ; ; in, for State estimation at time 10:00 For Kalman gain, for The observed value at time, For the observation matrix, Let be the covariance matrix. In this invention, the state transition matrix is ​​set as the identity matrix. (In other embodiments, an appropriate state transition matrix can be selected based on the physiological signal model.) A ), observation matrix It can be dynamically adjusted based on a model of heart rate and respiratory rate.

[0060] The adaptive physiological signal filtering technique described in the above embodiments processes physiological signals (such as heart rate and respiratory rate) acquired by millimeter-wave radar. Compared with the performance degradation problem of traditional fixed-parameter filtering methods in motion interference scenarios, the present invention proposes an adaptive physiological signal filtering technique with the following specific improvements: I. Multi-level Filtering Architecture: This invention employs a multi-level filtering architecture, comprising three levels: primary noise removal, nonlinear signal modeling, and adaptive parameter updating. Primary noise removal primarily targets high-frequency interference and baseline drift; nonlinear signal modeling processes the nonlinear characteristics of physiological signals using an extended Kalman filter (EKF); and adaptive parameter updating adjusts the filtering parameters according to the dynamic scene.

[0061] II. Improved Extended Kalman Filter: Addressing the nonlinear characteristics of physiological signals, this invention extends the traditional Kalman filter by linearizing the nonlinear function through Taylor expansion. The state prediction and update equations of the extended Kalman filter are as follows: Prediction steps: ; ; Update steps: ; ; ; in, and These are the nonlinear state transition function and the observation function, respectively. and This is the corresponding Jacobian matrix. and These are the covariance matrices of process noise and observation noise, respectively.

[0062] III. Innovation in Variational Bayesian Filtering (VBF): For scenarios with strong interference or complex motion, this invention introduces variational Bayesian filtering technology. Its core idea is to infer an approximate posterior distribution through variational methods and minimize the KL divergence through iterative optimization. Compared with traditional filtering methods, VBF has the following advantages: 1. It can handle non-Gaussian noise and multimodal distribution, and adapt to complex interference environments.

[0063] 2. Automatically learns noise model parameters, eliminating the need to manually set the noise covariance.

[0064] 3. Provides a measure of uncertainty in state estimation, enhancing the reliability of decision-making.

[0065] IV. Adaptive Noise Estimation: This invention implements an adaptive estimation algorithm for the noise covariance matrix. By analyzing the statistical characteristics of the innovative sequence (observation residuals), the noise model parameters are dynamically adjusted. Specifically, when motion interference is detected (judged by accelerometer readings or signal rate of change), the process noise covariance is automatically increased to improve the filter's adaptability; conversely, the process noise covariance is decreased in a static state to improve filtering accuracy.

[0066] Through the aforementioned adaptive physiological signal filtering technique, this invention achieves a significant performance improvement in processing millimeter-wave radar physiological signals under motion interference scenarios. Experiments show that under mild motion interference, the root mean square error (RMSE) of heart rate detection is reduced by approximately 40%.

[0067] In one embodiment, the noise characteristics of the signal may be more complex in scenarios with strong interference or motion. Therefore, variational Bayesian filtering can be used to optimize the state distribution by dynamically adjusting the prior and posterior distributions and minimizing the KL divergence, thereby reducing the root mean square error (RMSE) of heart rate detection, improving anti-interference capability, and obtaining more reliable target physiological filtering features.

[0068] Furthermore, based on convolutional neural networks and self-attention mechanisms, feature extraction is performed on the target video feature information to obtain the target facial expression features and target body posture features. In this invention, a high-resolution camera can be used to capture information such as facial expressions, eye movements, and body posture.

[0069] In this invention, the facial expression recognition process can use ResNet-50 or a similar CNN backbone network combined with a self-attention mechanism, with an input size of 224×224 pixels and an output facial feature vector. The body pose detection process can obtain the positions of the main joints of the human body through keypoint detection algorithms (such as OpenPose), and then use a Graph Convolutional Network (GCN) to analyze the spatiotemporal relationships between joints. It can be represented as: ; in, For the graph adjacency matrix, For node features, For learnable weights, For example, ReLU is an activation function.

[0070] Furthermore, in this invention, CNNs automatically learn local patterns and features in images through structures such as convolutional layers and pooling layers. When processing target video feature information (such as facial expressions, body postures, etc.), CNNs can capture subtle changes in facial features, the position and angle of body joints, and other information. For example, through convolutional operations, features such as the degree of eye opening and closing and the shape of the mouth can be extracted, which are closely related to facial expressions.

[0071] Self-attention mechanisms enable models to focus more on key areas related to emotions when processing video information. In facial expression recognition, self-attention mechanisms can automatically focus on key areas such as the eyes and mouth, ignoring irrelevant information such as the background, thereby improving the accuracy of feature extraction. For body posture feature extraction, self-attention mechanisms help models focus on key body joints and capture the connection between posture changes and emotions. After processing by CNN and self-attention mechanisms, the resulting target facial expression features can accurately describe subtle changes in facial expressions, such as feature vectors corresponding to smiles and frowns. Target body posture features reflect the position and posture information of various body parts, such as arm swings and body tilts, which are also related to emotional states. Furthermore, in environments with changing lighting or partial occlusion, self-attention mechanisms can help the network focus on key facial areas, improving recognition accuracy; at the same time, when fused with other modal information, it can compensate for the loss of recognition accuracy when video modalities fail.

[0072] Furthermore, the features extracted from different modalities (voice, physical characteristics, and video) are integrated to construct the first multimodal data. The fusion of multimodal data provides more comprehensive and richer information because data from different modalities reflects the user's emotional state from different perspectives. For example, voice features may include emotional information such as tone and speech rate, physical characteristics reflect physiological emotional responses, and facial expressions and body postures directly demonstrate the outward manifestation of emotions. By integrating this information, the user's emotional state can be analyzed and judged more accurately.

[0073] Additionally, a second multimodal dataset can be constructed based on target emotion category features and target physiological filtering features. Compared to the first multimodal dataset, the second multimodal dataset only includes target emotion category features and target physiological filtering features, excluding video-related features. This data construction method is used in scenarios with high privacy requirements or limited access to video data. For example, in some medical monitoring or psychological counseling scenarios, to protect user privacy, only voice and vital sign information is used for emotion analysis, avoiding the collection and processing of video data. Although the information richness of the second multimodal dataset is relatively low, it can still reflect the user's emotional state to some extent.

[0074] Based on the above embodiments, the full-modal emotion state detection model is trained through the following steps: Based on the sample emotion category features, sample physiological filtering features, sample facial expression features, and sample body posture features, sample multimodal data corresponding to each modality is constructed. The sample multimodal data is a set of three nodes, including temporal frame, spatial location, and modal features. The various sample multimodal data are connected by hyperedges. The sample multimodal data is labeled with corresponding emotion state tags to obtain the training sample set; Based on the training sample set, the cross-modal spatiotemporal graph neural network is trained to obtain the full-modal emotion state detection model.

[0075] In this invention, the emotion category features of the samples are derived from the analysis of speech signals, such as features extracted through short-time Fourier transform and Mel frequency cepstral coefficients, and then processed by convolutional neural networks and bidirectional long short-term memory networks to obtain emotion category-related features.

[0076] The physiological filtering features of the samples are obtained by denoising physiological signals (such as heart rate and respiratory rate) using Kalman filtering, extended Kalman filtering, or variational Bayesian filtering.

[0077] Facial expression features can be extracted from video images using convolutional neural networks and self-attention mechanisms; while body pose features can be obtained by modeling body key point information in the video using graph convolutional networks.

[0078] Furthermore, the above features are mapped to a set of ternary nodes. ,in, Represents a time frame. Indicates spatial location, This method represents modal features, integrating information from different modalities into a unified graph structure. Each node contains information in three dimensions: temporal, spatial, and modal. In this invention, the multimodal data samples are connected by hyperedges, which contain cross-modal spatiotemporal correlations, i.e., correlation information between different modalities, different temporal frames, and different spatial locations. This connection method can capture the complex relationships between multimodal data, providing a foundation for subsequent graph convolution calculations.

[0079] Furthermore, each sample of multimodal data is labeled with a corresponding emotional state tag, such as happy, sad, and anxious. These tags are used to supervise the training of the model, enabling the cross-modal spatiotemporal graph neural network to learn the mapping relationship between different emotional states and multimodal features. Then, all the multimodal data samples labeled with emotional state tags are combined into a training sample set, which will be used for subsequent training of the cross-modal spatiotemporal graph neural network.

[0080] In graph convolution computation, this invention learns the weighting coefficients of each mode at different times and spaces through a differentiable gating mechanism. The dynamic graph convolution operator is based on nodes (such as...) and The spatiotemporal connectivity and corresponding temporal distance between ) ), and modal adaptive weight matrix ( The node features are updated. Specifically, the process of training the cross-modal spatiotemporal graph neural network based on the training sample set to obtain the full-modal emotion state detection model includes: Based on the differentiable gating mechanism, the weight matrix in the dynamic graph convolution operator of the cross-modal spatiotemporal graph neural network is adjusted. After determining that the training results meet the preset conditions, the full-modal emotion state detection model is obtained. The dynamic graph convolution operator is: ; in, and These are nodes corresponding to different samples of multimodal data. In multimodal sentiment analysis, and For data nodes corresponding to different modalities, for example, Nodes that may represent speech modalities, and Nodes that may represent facial expression modalities.

[0081] For nodes In the cross-modal spatiotemporal graph neural network, the first The feature representation of the layer is the output result to be calculated, i.e., the node. The updated features.

[0082] Nonlinear activation functions (such as ReLU) are used to introduce nonlinear transformations and enhance the expressive power of the model.

[0083] Represents nodes The set of all connected hyperedges: In a cross-modal graph structure, a hyperedge can connect nodes of multiple different modalities, thereby capturing cross-modal relationships.

[0084] For super-edge The set of all connected nodes; Indicates the superedge The number of connected nodes is used to normalize the contributions of different nodes.

[0085] This indicates the first time after adjustment by the differentiable gating mechanism. The first layer The weight matrix corresponding to each modality is used to reflect the importance of different modalities in the graph convolution process. For example, in some scenarios, speech may be more important than facial expressions, so the weight of the speech modality will be greater.

[0086] For nodes In the The feature vector representation of a layer is the input feature passed into the current computation process.

[0087] For nodes With nodes The temporal distance between the nodes (representing the relative positions of the two nodes on the time axis) is a parameter that captures the relationship between different modalities of data in the time dimension. For example, there may be a time difference between emotional expression in speech and facial expressions, and this parameter can model this temporal dependency.

[0088] This indicates a feature concatenation operation, used to combine node features. Distance between time series By combining these features, richer feature representations are generated, enabling the model to consider both node features and temporal relationships simultaneously.

[0089] This invention, through the aforementioned dynamic graph convolution operator, can adaptively learn the contribution of different modalities to emotional state detection under different time and space conditions, thereby achieving more accurate multimodal emotion perception.

[0090] Furthermore, multimodal feature fusion is achieved using a deep neural network (DNN). In this invention, a cross-modal transformation network and a multimodal self-attention mechanism are utilized to calculate the attention weights of each modality's features, thereby fusing features from different modalities. The formula corresponding to the cross-modal transformation network is as follows: ; in, This represents the fused feature vector, obtained by weighted summation of the features of each modality; Indicates the first The feature vectors of the three modalities are: speech, physical signs and video data. Indicates the first Weights of each modality.

[0091] Furthermore, attention weights are calculated using a multimodal self-attention mechanism. The formula is: ; in, This represents the feature vector after processing by the self-attention mechanism. Indicates the first 1 eigenvector.

[0092] Furthermore, the cross-modal spatiotemporal graph neural network is trained using a training sample set. During training, the model adjusts its parameters using backpropagation to ensure that the predicted emotion state outputs are as consistent as possible with the actual emotion state labels. After training, a full-modal emotion state detection model is obtained. In this invention, the cross-modal spatiotemporal graph neural network is an innovative neural network structure, and its main differences from traditional dynamic graph neural networks are as follows: I. Unified 3D Modeling: Cross-modal spatiotemporal graph neural networks unify the temporal, spatial, and modal dimensions into a third-order hypergraph structure, where each node is represented as a triple ( t , s , m ),in, t Represents a time frame. s Indicates spatial location, m This represents modal features. This unified three-dimensional modeling approach enables the network to simultaneously capture the temporal and spatial correlations of different modal data.

[0093] II. Cross-modal Hyperedge Connections: Unlike traditional graph structures where edges connect only two nodes, the cross-modal hyperedges in this invention can connect multiple nodes simultaneously. These nodes may come from different modalities, different points in time, or different spatial locations. Hyperedge connection structures can effectively represent complex relationships between multimodal data, such as the temporal correlation between speech and facial expressions, and the mutual influence between physiological signals and body movements.

[0094] III. Differentiable Gating Mechanism: This invention introduces a differentiable gating mechanism to automatically learn the importance weights of each modality under different spatiotemporal conditions. This mechanism can dynamically adjust the contribution of different modalities based on the characteristics of the input data, making the network more flexible and accurate in processing complex multimodal spatiotemporal data. For example, in cases of poor video quality, the weight of the video modality may be automatically reduced, while the weights of the speech and physiological signal modalities may be increased.

[0095] The cross-modal spatiotemporal graph neural network structure provided by this invention can effectively solve the modal fragmentation problem in traditional multimodal fusion methods, capture richer cross-modal spatiotemporal dependencies, and thus improve the accuracy and robustness of emotion state detection.

[0096] In this invention, the full-modal emotion state detection model can comprehensively consider information from multiple modalities such as speech, physiology, facial expressions, and body posture, and more accurately detect the user's emotional state. This model has broad application prospects in fields such as mental health monitoring, human-computer interaction, and emotion computing.

[0097] Based on the above embodiments, the privacy modality emotion state detection model is obtained through the following steps: The full-modal emotion state detection model is used as the teacher model; Based on knowledge distillation, the attention map and hidden layer output features in the teacher model are transferred to the student network to obtain the privacy modality emotion state detection model. The student network is obtained by training the cross-modal spatiotemporal graph neural network based on the sample time-frequency features and the sample physiological filtering features.

[0098] In this invention, the full-modal emotion state detection model is a model capable of integrating information from multiple modalities (such as voice, vital signs, facial expressions, body posture, etc.) to detect emotion states. In scenarios with low privacy requirements or where user authorization is required, the full-modal emotion state detection model can utilize video information to improve recognition accuracy.

[0099] The full-modal emotion state detection model, serving as a teacher model, contains rich attention features and intermediate representations. Knowledge distillation is a model compression and transfer learning technique designed to transfer knowledge from large, complex teacher models to small, simple student models.

[0100] In this invention, the teacher model is a full-modal emotion state detection model, while the student model is a privacy-modal emotion state detection model. The attention map in the teacher model reflects the model's focus when processing information from different modalities. By transferring the attention map, the student model can learn how to effectively focus on information more important for emotion state recognition.

[0101] The hidden layer output features of the teacher model contain the intermediate representations of the model when processing input data. These intermediate representations play an important role in the recognition of emotional states. By transferring these features, the student model can learn the feature extraction and representation capabilities of the teacher model when processing data.

[0102] In this invention, the privacy modality emotion state detection model uses only information that does not involve video, such as speech and millimeter-wave radar, to detect emotions, so as to avoid potential privacy issues caused by video. Therefore, the student network is obtained by training a cross-modal spatiotemporal graph neural network based on sample time-frequency features and sample physiological filtering features.

[0103] During the training of the student network, the attention map and hidden layer output features of the teacher model are transferred to the student network through distillation of the loss function. Specifically, this invention employs hierarchical feature transfer. Unlike traditional knowledge distillation, which only transfers knowledge from the final output layer, this invention uses progressive feature distillation to perform multi-level knowledge transfer from low-level feature representations to high-level semantic features. That is, knowledge is transferred from all levels of the full-modal teacher network (including shallow low-level features, mid-level structural features, and deep semantic features) to the privacy-modal student network, enabling the student network to learn the feature representation capabilities of the teacher network more comprehensively.

[0104] Furthermore, this invention focuses on the attention distribution mechanism in a multimodal teacher network, transferring the attention map formed by the teacher network when processing multimodal data to the student network. The attention map reflects which parts of the input data the network focuses on. By transferring these attention patterns, the student network can learn how to still focus on key information in emotional state judgment even in the absence of video modality.

[0105] Furthermore, this invention employs a dynamic weighting mechanism to balance the loss contribution of feature distillation at each layer, and the distillation loss function... L distill Specifically: in, LCE For classification cross-entropy loss, and The teacher network and the student network are respectively l Layer feature representation, and These are attention graphs for the teacher network and the student network, respectively. Let wl be the feature alignment function, and wl be the dynamic weights of the l-th layer. α , β and γ Hyperparameters are used to balance various types of losses.

[0106] During the training process, this invention adopts a progressive training strategy, first using high-level semantic features for distillation, and then gradually introducing distillation of mid-level and low-level features. This strategy can avoid the interference of low-level features on the learning process and improve the efficiency and effectiveness of knowledge transfer.

[0107] Through the aforementioned progressive feature distillation technique, this invention achieves a privacy-preserving modality emotion detection model that maintains near-full-modality recognition accuracy even without using video modality. In other words, the student network can still learn effective feature representations and attention allocation mechanisms from the teacher model without using video information, thus maintaining a high accuracy rate in emotion recognition. In experimental evaluations, the privacy-preserving model's recognition accuracy is only 2 to 3 percentage points lower than the full-modality model, but it completely avoids the privacy risks associated with video data, providing an ideal solution for highly privacy-sensitive scenarios (such as medical and psychological counseling).

[0108] In this invention, the emotion state detection method provided by this invention is applied to a corresponding emotion state detection system, enabling the emotion state detection system to achieve real-time computation on NVIDIA Jetson or embedded GPU platforms, with a typical frame rate of over 30fps. The response latency for the privacy modality is approximately 120ms, and for the full modality, it is approximately 180ms, which meets the requirements of most interaction scenarios. The emotion state detection system of this invention can be applied to various scenarios, including but not limited to: I. Mental Health Monitoring Scenarios: The emotion state detection system can be used in conjunction with clinical scales (such as the PHQ-9 health questionnaire) to help determine an individual's risk level of depression or anxiety through dynamic long-term monitoring of their emotion state. Furthermore, the emotion state detection system can generate emotion state trend reports, providing objective assessment data for mental health professionals, enabling early warning and guiding high-risk individuals to seek timely medical attention, thereby reducing the incidence of serious mental illnesses.

[0109] II. Intelligent Cockpit and Human-Machine Interaction Scenarios: In the in-vehicle environment, the emotion state detection system can identify the driver's emotional state in real time. When it detects that the driver is in a risky emotional state such as obvious fatigue, anger, or severe distraction, the emotion state detection system can trigger a corresponding safety reminder mechanism. In the field of human-machine interaction, robots or intelligent assistants can dynamically adjust interaction strategies and service content according to the user's emotional state, providing a more personalized intelligent service experience.

[0110] III. Application in Educational Scenarios: In distance education or intelligent learning systems, the emotion state detection system can monitor changes in learners' emotional states. When it detects that learners are confused, bored, or lacking concentration, the intelligent teaching system can adjust the teaching content, difficulty, or presentation method accordingly to improve learning effectiveness and experience.

[0111] IV. Medical Rehabilitation Assistance: In the process of medical rehabilitation, the emotional state detection system can be used to monitor the patient's emotional changes during treatment and rehabilitation, providing medical staff with an objective reference for the patient's psychological state and assisting in the development of more effective rehabilitation plans.

[0112] The above application scenarios fully demonstrate the advantages of this invention in terms of real-time performance, privacy protection, and multi-scenario adaptability. Through the dual-path architecture of privacy modality and full modality, the emotion state detection system can flexibly select appropriate modality combinations according to the privacy requirements of different scenarios, while maintaining high emotion recognition accuracy.

[0113] This invention transfers effective features from a full-modal emotion state detection model to a privacy-modal emotion state detection model through knowledge distillation, which can maintain a high accuracy rate in emotion recognition while protecting user privacy.

[0114] The emotion state detection system provided by the present invention is described below. The emotion state detection system described below can be referred to in correspondence with the emotion state detection method described above.

[0115] Figure 2 This is a schematic diagram of the structure of the emotion state detection system provided by the present invention, as shown below. Figure 2 As shown, this invention provides an emotion state detection system, including a multimodal data acquisition module 201 and an emotion state detection module 202. The multimodal data acquisition module 201 acquires target multimodal data corresponding to the user to be detected. The emotion state detection module 202 inputs the target multimodal data into a multimodal emotion state detection model to obtain the emotion detection result of the user to be detected output by the multimodal emotion state detection model. The multimodal emotion state detection model is obtained by training a cross-modal spatiotemporal graph neural network based on sample multimodal data. The cross-modal spatiotemporal graph neural network is trained using a differentiable gating mechanism to obtain the weight coefficients corresponding to each modality.

[0116] The emotion state detection system provided by this invention acquires target multimodal data of the user to be detected. This data covers information from different modalities and together constitutes the data foundation for emotion state analysis. Then, the acquired target multimodal data is input into a multimodal emotion state detection model, which outputs the emotion detection result of the user to be detected. Since the emotion detection result integrates the information in the multimodal data and considers the spatiotemporal dynamic characteristics and importance weights between different modal data, it can more accurately reflect the user's true emotional state.

[0117] The system provided in this embodiment of the invention is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0118] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 3As shown, the electronic device may include: a processor 301, a communications interface 302, a memory 303, and a communication bus 304. The processor 301, communications interface 302, and memory 303 communicate with each other via the communication bus 304. The processor 301 can call logical instructions in the memory 303 to execute an emotion state detection method. This method includes: acquiring target multimodal data corresponding to the user to be detected; inputting the target multimodal data into a multimodal emotion state detection model to obtain the emotion detection result of the user to be detected output by the multimodal emotion state detection model. The multimodal emotion state detection model is obtained by training a cross-modal spatiotemporal graph neural network based on sample multimodal data. The cross-modal spatiotemporal graph neural network is trained using a differentiable gating mechanism to obtain the weight coefficients corresponding to each modality.

[0119] Furthermore, the logical instructions in the aforementioned memory 303 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0120] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the emotion state detection method provided by the above methods, the method comprising: acquiring target multimodal data corresponding to a user to be detected; inputting the target multimodal data into a multimodal emotion state detection model to obtain the emotion detection result of the user to be detected output by the multimodal emotion state detection model, wherein the multimodal emotion state detection model is obtained by training a cross-modal spatiotemporal graph neural network based on sample multimodal data; the cross-modal spatiotemporal graph neural network is trained to obtain the weight coefficients corresponding to each modality through a differentiable gating mechanism.

[0121] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the emotion state detection method provided in the above embodiments. The method includes: acquiring target multimodal data corresponding to a user to be detected; inputting the target multimodal data into a multimodal emotion state detection model to obtain the emotion detection result of the user to be detected output by the multimodal emotion state detection model, wherein the multimodal emotion state detection model is obtained by training a cross-modal spatiotemporal graph neural network based on sample multimodal data; the cross-modal spatiotemporal graph neural network is trained to obtain the weight coefficients corresponding to each modality through a differentiable gating mechanism.

[0122] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0123] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting emotional states, characterized in that, include: Obtain the target multimodal data corresponding to the user to be detected; The target multimodal data is input into a multimodal emotion state detection model to obtain the emotion detection result of the user to be detected output by the multimodal emotion state detection model. The multimodal emotion state detection model is obtained by training a cross-modal spatiotemporal graph neural network based on sample multimodal data. The cross-modal spatiotemporal graph neural network is trained to obtain the weight coefficients corresponding to each modality through a differentiable gating mechanism.

2. The emotional state detection method according to claim 1, characterized in that, The multimodal emotion state detection model includes a full-modal emotion state detection model and a privacy-modal emotion state detection model; the target multimodal data includes first multimodal data and second multimodal data; The step of inputting the target multimodal data into a multimodal emotion state detection model to obtain the emotion detection result of the user to be detected output by the multimodal emotion state detection model includes: The first multimodal data is input into the full-modal emotion state detection model to obtain the emotion detection result of the user to be detected output by the full-modal emotion state detection model. The first multimodal data is constructed based on the target speech feature signal, target vital sign signal and target video feature information of the user to be detected. The target vital sign signal is obtained based on millimeter-wave radar. Alternatively, the second multimodal data can be input into the privacy modality emotion state detection model to obtain the emotion detection result of the user to be detected output by the privacy modality emotion state detection model, wherein the second multimodal data is constructed based on the target speech feature signal and the target vital sign signal of the user to be detected.

3. The emotional state detection method according to claim 2, characterized in that, Before inputting the target multimodal data into the multimodal emotion state detection model to obtain the emotion detection result of the user to be detected output by the multimodal emotion state detection model, the method further includes: Based on the short-time Fourier transform and Mel frequency cepstral coefficients, the target time-frequency features corresponding to the target speech feature signal are obtained; The target time-frequency features are input into the speech feature extraction model to obtain the target emotion category features output by the speech feature extraction model. The speech feature extraction model is constructed based on a convolutional neural network and a bidirectional long short-term memory network. Based on Kalman filtering, extended Kalman filtering, or variational Bayesian filtering, the target vital sign signal is denoised to obtain the target physiological filtered features. Based on convolutional neural networks and self-attention mechanisms, feature extraction is performed on the target video feature information to obtain target facial expression features and target body posture features; The first multimodal data is constructed based on the target emotion category features, the target physiological filtering features, the target facial expression features, and the target body posture features. The second multimodal data is constructed based on the target emotion category features and the target physiological filtering features.

4. The emotional state detection method according to claim 3, characterized in that, The full-modal emotion state detection model was trained through the following steps: Based on the sample emotion category features, sample physiological filtering features, sample facial expression features, and sample body posture features, sample multimodal data corresponding to each modality is constructed. The sample multimodal data is a set of three nodes, including temporal frame, spatial location, and modal features. The various sample multimodal data are connected by hyperedges. The sample multimodal data is labeled with corresponding emotion state tags to obtain the training sample set; Based on the training sample set, the cross-modal spatiotemporal graph neural network is trained to obtain the full-modal emotion state detection model.

5. The emotional state detection method according to claim 4, characterized in that, The privacy-preserving modality emotion state detection model is obtained through the following steps: The full-modal emotion state detection model is used as the teacher model; Based on knowledge distillation, the attention map and hidden layer output features in the teacher model are transferred to the student network to obtain the privacy modality emotion state detection model. The student network is obtained by training the cross-modal spatiotemporal graph neural network based on the sample time-frequency features and the sample physiological filtering features.

6. The emotional state detection method according to claim 4, characterized in that, The process of training the cross-modal spatiotemporal graph neural network based on the training sample set to obtain the full-modal emotion state detection model includes: Based on the differentiable gating mechanism, the weight matrix in the dynamic graph convolution operator of the cross-modal spatiotemporal graph neural network is adjusted. After determining that the training results meet the preset conditions, the full-modal emotion state detection model is obtained. The dynamic graph convolution operator is: ； in, and These are the nodes corresponding to different samples of multimodal data. For nodes In the cross-modal spatiotemporal graph neural network, the first Layer feature representation, It is a non-linear activation function. Represents nodes The set of all connected superedges. For super-edge The set of all connected nodes. Indicates the superedge Number of connected nodes; This indicates the first time after adjustment by the differentiable gating mechanism. The first layer The weight matrix corresponding to each mode; For nodes In the The feature vector representation of the layer, For nodes With nodes The time distance between them This indicates a feature splicing operation.

7. An emotion state detection system, characterized in that, include: The multimodal data acquisition module is used to acquire the target multimodal data corresponding to the user to be detected; The emotion state detection module is used to input the target multimodal data into the multimodal emotion state detection model to obtain the emotion detection result of the user to be detected output by the multimodal emotion state detection model. The multimodal emotion state detection model is obtained by training a cross-modal spatiotemporal graph neural network based on sample multimodal data. The cross-modal spatiotemporal graph neural network is trained to obtain the weight coefficients corresponding to each modality through a differentiable gating mechanism.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the emotional state detection method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the emotional state detection method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the emotional state detection method as described in any one of claims 1 to 6.

Images