A method and system for assessing emotion state based on multiple events and multiple tasks.

Abstract

This invention provides a method and system for assessing emotional state based on multiple events and tasks, relating to the fields of mental health assessment and artificial intelligence. The method specifically includes: collecting physiological data of the assessment subject and conducting psychological tests on the subject; generating emotional state labels for the subject based on the psychological test results; constructing and fusing a neural network model of electroencephalogram (EEG) signals and an electrical myocardiogram (EMG) signal neural network model to obtain a multimodal fusion model; constructing a multi-task classifier based on the emotional state labels; inputting the physiological data of the assessment subject into the multimodal fusion model for feature fusion, generating a multimodal joint feature representation, and inputting this representation into the multi-task classifier for assessment to obtain the emotional state assessment result for the subject. This invention overcomes the shortcomings of insufficient accuracy in analyzing single data sources, enabling a more accurate and comprehensive assessment of the mental health status of university students, and providing strong support for the early detection and intervention of mental health problems such as depression.

Description

Technical Field

[0001] This invention relates to the fields of mental health assessment and artificial intelligence technology, and in particular to a method and system for assessing emotional states based on multiple events and multiple tasks. Background Technology

[0002] In today's fast-paced and highly competitive society, the mental health of university students is receiving increasing attention. Mental health issues such as depression, anxiety, and social phobia not only affect their learning efficiency and quality of life, but can also lead to a series of serious consequences, such as extreme behaviors. However, early detection and timely intervention for mental health problems among university students is an extremely challenging task. Traditional mental health assessment methods have many shortcomings, such as being time-consuming and labor-intensive, relying on subjective reports, and being limited by medical resources, making it difficult to meet the needs for efficient and accurate assessment.

[0003] With the rapid development of artificial intelligence technology, its application in the field of college student mental health assessment has broad prospects. Deep learning models can objectively analyze multi-source data from multiple dimensions, comprehensively capturing subtle clues of an individual's emotions, feelings, and psychological state, thus improving the accuracy of the assessment. However, existing deep learning-based mental health assessment methods rely too heavily on explicit information such as behavioral, audio, and video data. These data reflect a person's outward performance and cannot directly reflect their internal cognitive state and emotional activity, resulting in limitations such as inconsistency between outward performance and internal state, lack of in-depth analysis of cognitive processes, and insufficient expression of emotional information. Summary of the Invention

[0004] To address the shortcomings of the existing technologies, this invention proposes a multi-event, multi-task-based method and system for assessing emotional states by comprehensively analyzing electroencephalogram (EEG) and electromyogram (EMG) signals. The aim is to further improve the accuracy of the analysis by obtaining more objective, comprehensive, and reliable information, thereby helping college students identify potential mental health problems early and take effective measures to address and improve them.

[0005] The first aspect of this invention proposes a method and system for assessing emotional states based on multiple events and multiple tasks. The method includes the following steps:

[0006] Step 1: Collect physiological data of the assessment subjects and conduct psychological tests on them. Generate emotional state labels for the assessment subjects based on the psychological test results.

[0007] Step 2: Preprocess the physiological data of the evaluation subjects and construct a multimodal dataset, and divide the multimodal dataset into training set and test set according to the set ratio;

[0008] Step 3: Construct neural network models for electroencephalogram (EEG) signals and electromyography (EMG) signals, respectively;

[0009] Step 4: Fuse the EEG neural network model and the EMG neural network model to obtain a multimodal fusion model;

[0010] Step 5: Construct a multi-task classifier based on emotion state labels;

[0011] Step 6: Iteratively train the multimodal fusion model and multi-task classifier using the training set. Stop iterating when the total loss function of the multi-task classifier converges, reaches the preset number of training rounds, or meets the preset classification target, and obtain the trained multimodal fusion model and multi-task classifier.

[0012] Step 7: Validate the trained multimodal fusion model and multi-task classifier using the test set to obtain the final multimodal fusion model and multi-task classifier;

[0013] Step 8: For any assessment object, collect the physiological data of the assessment object and input it into the final multimodal fusion model for feature fusion. Then, input the generated multimodal joint feature representation into the multi-task classifier for evaluation to obtain the emotional state assessment result of the assessment object.

[0014] The method for collecting physiological data in step 1 is as follows: multimodal physiological and behavioral tests are performed on the assessment subject, and the physiological data of the assessment subject is recorded in real time; wherein the process of multimodal physiological and behavioral testing is as follows: the assessment subject is fitted with an EEG sensor and an EMG sensor to record the EEG signal and EMG signal of the assessment subject respectively, and the EEG signal and EMG signal are used as the physiological data of the assessment subject.

[0015] Several behavioral tests were conducted on the subjects in sequence, and their physiological data were recorded in real time during each test. Each behavioral test consisted of one of the following: a self-introduction test, a text reading test, or a gait test.

[0016] The self-introduction test involves the subject introducing themselves based on a given theme; the text reading test involves the subject reading aloud a designated text; and the gait test involves the subject walking orderly within a designated area according to a predetermined route and direction.

[0017] The psychological tests were conducted using a personality scale, a general well-being scale, a depression screening scale, and a depressive symptom scale. The scores of each scale were used as labels for the emotional state of the subjects.

[0018] The emotional state labels include: the five personality dimensions of the assessment subject and the psychological state of the assessment subject;

[0019] Step 2 further includes:

[0020] Step 2.1: Based on the timestamp information recorded in the physiological data, mark the start and end times of the assessment subject's different activity stages, and divide the physiological data into several segments according to the marked start and end times to obtain the segmented physiological data;

[0021] Step 2.2: Downsample the segmented physiological data to obtain downsampled physiological data;

[0022] Step 2.3: Filter the downsampled physiological data to obtain filtered physiological data; a fourth-order Butterworth filter is used to bandpass filter the EEG signal in the physiological data; a bandpass filter is used to bandpass filter the EMG signal in the physiological data.

[0023] Step 2.4: Extract features from the filtered physiological data to obtain the time-domain and frequency-domain features of the physiological data;

[0024] Step 2.5: For any physiological data, combine the time-domain features and frequency-domain features of the physiological data into a feature vector of the physiological data. Take a behavioral test as an event, and take the feature vectors of all physiological data in each event as the multimodal data sequence of that event. Use the multimodal data sequences of all events to construct a multimodal dataset; wherein the multimodal dataset includes: electroencephalogram (EEG) dataset and electromyogram (EMG) dataset;

[0025] Step 2.6: Divide the multimodal dataset into training and test sets according to a preset ratio;

[0026] The EEG neural network model and the EMG neural network model mentioned in step 3 are both single-modality neural network models.

[0027] The single-modal neural network model is: using a convolutional neural network (CNN) model to extract local features from the input multimodal data sequence, wherein the convolutional neural network (CNN) model is a single-layer convolution operation;

[0028] The feature representation extracted by the Convolutional Neural Network (CNN) model is input into a multi-head self-attention mechanism for processing. An attention score is calculated for each physiological data point in the multimodal data sequence. This attention score is then used to adjust the value vector V. i Weighted summation is performed to obtain a weighted feature representation of the physiological data;

[0029] For any event, a residual connection is formed between the weighted feature representation of the physiological data in the event and the multimodal data sequence of the event. The result of the residual connection is then normalized by layer. The normalized features are then transformed nonlinearly through a feedforward network to generate the feature sequence of the event.

[0030] The feature sequence of the event is input into a bidirectional long short-term memory network (Bi-LSTM). The Bi-LSTM is used to process the forward and reverse time dependencies in the feature sequence, and the hidden state of the last time step in the feature sequence is taken as the output of the Bi-LSTM to obtain the feature vector of the event.

[0031] The process of fusing the EEG neural network model and the EMG neural network model described in step 4 is divided into two parts: a single-modal fusion process based on the channel attention module and a multi-modal fusion process based on the interactive attention mechanism.

[0032] The single-modal fusion process based on the channel attention module is as follows: a joint feature representation of multiple events is constructed using the feature vectors output by the EEG neural network model and the EMG neural network model, respectively. Global average pooling and global max pooling are then performed on the joint feature representation of multiple events. The obtained global average pooling features and global max pooling features are then subjected to dimensionality reduction and activation processing through a two-layer fully connected network to obtain the attention weights of the events. These weights are then fused with the joint feature representation of multiple events to generate a weighted multi-event feature representation.

[0033] For the EEG neural network model, the weighted multi-event feature representation obtained by fusing the feature vector output by the model with single-modal characteristics serves as the multi-event fusion feature H of the EEG signal. EEG For the EMG neural network model, the weighted multi-event feature representation obtained by fusing the feature vector output by the model with single-modal characteristics serves as the multi-event fusion feature H of the EMG signal. EMG ;

[0034] The multimodal fusion process based on the interactive attention mechanism is as follows: a dual-branch cross-attention module is used to calculate the multi-event fusion features H of the EEG signals. EEG Multi-event fusion features of electromyographic signals H EMG The feature correlation between them is used to obtain the attention output from the EEG mode to the EMG mode. Attention output from EMG mode to EEG mode Then through and The data is then spliced ​​together to generate separate EEG features. and electromyographic characteristics And integrates multimodal joint feature representations;

[0035] Furthermore, the specific content of the single-modal fusion process based on the channel attention module is as follows:

[0036] The feature vectors of each event in the multimodal dataset are fused along the channel dimension to construct a joint feature representation of the multiple events;

[0037] The joint feature representation of multiple events is subjected to global average pooling and global max pooling respectively to obtain the global average pooling feature H. t,avg and global max pooling feature H t,max The first layer of the fully connected network is used to process the global average pooling feature H. t,avg and global max pooling feature H t,max Dimensionality reduction is performed, and then a second fully connected network is used to activate the dimensionality-reduced global average pooling features and global max pooling features respectively. The activated global average pooling features and global max pooling features are then merged to obtain the attention weights of the events.

[0038] The attention weight of the event is represented as follows:

[0039] α event =Sigmoid(W1(W0(H) t,avg ))+W1(W0(H t,max )))

[0040] Where α event The attention weights for the events are represented by W1; the sigmoid function represents the activation function; W1 is the recovery matrix; and W0 is the dimensionality reduction matrix.

[0041] The attention weight α of the event is calculated using element-wise multiplication. event Joint feature representation of multiple events H t The features are fused to obtain a weighted multi-event feature representation H. event ;

[0042] Furthermore, the dual-branch cross-attention module is as follows: one attention branch uses the multi-event fusion feature H of EEG signals. EEG For querying the multi-event fusion features H of electromyographic signals EMG For a key and a value, calculate H. EEG With H EMG The correlation of features between them, and using H EEG With H EMG Calculate the attention output from the EEG mode to the EMG mode based on the feature correlation between them. Represented as:

[0043]

[0044] Where Softmax represents the normalized exponential function; This indicates the multi-event fusion feature H of EEG signals. EEG Mapped to the Query space; This indicates the multi-event fusion feature H of EEG signals. EEG Mapped to the Key space; V t (e→m) This indicates the multi-event fusion feature H of EEG signals. EEG Mapped to the Value space; e represents the EEG mode; m represents the EMG mode; d represents the feature dimension of the Query and Key, used to prevent the attention score from being too large in the high-dimensional space, which would lead to gradient vanishing or numerical instability;

[0045] Another attention branch is based on the multi-event fusion feature H of electromyographic signals. EMG For querying the multi-event fusion features H of EEG signals EEG For a key and a value, calculate H. EMG With H EEG The correlation of features between them, and using H EMG With H EEG Calculate the attention output from the EMG mode to the EEG mode based on the feature correlation between them. Represented as:

[0046]

[0047] in This indicates the multi-event fusion feature H of electromyographic signals. EMG Mapped to the Query space; This indicates the multi-event fusion feature H of electromyographic signals. EMG Mapped to the Key space; V t (m→e) This indicates the multi-event fusion feature H of electromyographic signals. EMG Mapped to the Value space;

[0048] The method for constructing the multi-task classifier in step 5 is as follows: Two classification tasks are defined based on emotional state labels. One classification task is a three-level classification problem of psychological states; this three-level classification problem is defined as a three-classification task, including three categories: normal, mild problems, and severe problems. The other classification task is a two-level classification problem of five personality types; this two-level classification problem of five personality types is defined as five two-classification tasks, including five categories: openness, conscientiousness, extraversion, agreeableness, and neuroticism.

[0049] Define a classifier for each classification task to evaluate the emotional state of the physiological data corresponding to the input multimodal joint feature representation based on the input multimodal joint feature representation;

[0050] The total loss function L of the multi-task classifier in step 6 total for:

[0051] L total =βL m +γ(L1+L2+L3+L4+L5)

[0052] Where L m β represents the cross-entropy loss of the three-level classification problem of mental states; L1, L2, L3, L4, and L5 represent the cross-entropy loss of the two-level classification problems of the five personality types; and γ represents the weight coefficients of the two-level classification problems of the five personality types.

[0053] Where L m The cross-entropy losses for L1, L2, L3, L4, and L5 are all expressed as:

[0054]

[0055] Where L represents the cross-entropy loss; N represents the total number of data in the training set; C represents the total number of classes; y lj Let be the true label of the l-th sample in the j-th category; This represents the evaluation result for the l-th sample;

[0056] The second aspect of this invention proposes an emotion state assessment system based on multiple events and multiple tasks, used to implement the emotion state assessment method based on multiple events and multiple tasks. The system includes: a data acquisition module, a data processing module, a feature fusion module, and a classification assessment module.

[0057] The data acquisition module is used to collect physiological data of the assessment subject and obtain emotional state labels of the assessment subject, and transmit the collected physiological data to the data processing module and transmit the obtained emotional state labels to the classification assessment module.

[0058] The data processing module is used to perform preprocessing on the received physiological data in sequence, including segmentation, downsampling, filtering, feature extraction and splicing, and to construct a multimodal dataset using the preprocessed physiological data, and then transmit the multimodal dataset to the feature fusion module.

[0059] The feature fusion module is used to extract features from different modalities in the multimodal dataset in a single modality, and to perform single-modal fusion based on the channel attention module and multimodal fusion based on the interactive attention mechanism on the extracted feature vectors in sequence, so as to generate a multimodal joint feature representation and transmit it to the classification evaluation module.

[0060] The classification and evaluation module is used to construct a classifier based on the emotional state label, and input the multimodal joint feature representation into the classifier to generate an emotional state evaluation result of the physiological data corresponding to the multimodal joint feature representation.

[0061] The beneficial effects of adopting the above technical solution are as follows:

[0062] 1) Improved objectivity and comprehensiveness: The method of this invention first obtains data on users' psychological state and emotional state labels such as personality through a questionnaire survey. Simultaneously, it utilizes wearable EEG and EMG sensors to record neural and muscle activity signals, forming multimodal data. By collecting and fusing multimodal information such as physiological data, this method can comprehensively and objectively assess the psychological state of university students, avoiding the limitations of single subjective tests.

[0063] 2) Enhanced Analysis Accuracy and Interpretability: The method of this invention preprocesses multimodal data to eliminate noise and extract features, then uses deep learning techniques to build separate unimodal models, and fuses the outputs of different unimodal models to construct a multimodal fusion model. This method utilizes deep learning techniques for feature representation learning and modeling of multimodal data, improving the accuracy of the analysis. Simultaneously, through interpretability techniques, such as gradient / attention visualization, the contribution of different modalities to the prediction results can be explained, improving the model's transparency.

[0064] 3) Improved evaluation efficiency and convenience: The method of this invention is based on a data-driven model evaluation approach, which reduces the workload of manual evaluation and improves the automation level and efficiency of the evaluation process. At the same time, it does not require the collection of private information, making it easy to promote and apply on a large scale.

[0065] 4) Basis for Mental Health Education and Intervention: Based on the analysis results output by the model, the method of this invention can understand the overall psychological state of university students, providing a basis for schools to formulate targeted mental health education content. Simultaneously, it can promptly identify and provide intervention and counseling for high-risk groups.

[0066] In summary, the method of this invention, by constructing a multimodal fusion model, can comprehensively analyze the psychological state and personality traits of college students and deeply explore the intrinsic relationship between multimodal data and mental health status. This method overcomes the shortcomings of insufficient accuracy in analyzing single data sources, enabling a more accurate and comprehensive assessment of college students' mental health. It provides strong support for early detection of psychological problems and subsequent intervention, and has broad application prospects in the fields of mental health and healthcare. Attached Figure Description

[0067] Figure 1 This is a flowchart of an emotion state assessment method based on multiple events and multiple tasks in this embodiment;

[0068] Figure 2 This is a diagram of the experimental protocol for multimodal physiological and behavioral testing in this embodiment;

[0069] Figure 3 This is a diagram illustrating the construction of the multimodal fusion model in this embodiment;

[0070] Figure 4 This is a structural diagram of an emotion state assessment system based on multiple events and multiple tasks in this embodiment. Detailed Implementation

[0071] To facilitate understanding of this application, specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and embodiments. The following embodiments are illustrative of the invention but are not intended to limit its scope. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application.

[0072] This embodiment provides a method for assessing emotional states based on multiple events and multiple tasks, such as... Figure 1 As shown, the method includes the following steps:

[0073] Step 1: Collect physiological data of the assessment subjects and conduct psychological tests on them. Generate emotional state labels for the assessment subjects based on the psychological test results.

[0074] In this embodiment, a multimodal data acquisition method is used to comprehensively record the user's physiological and behavioral responses in various situations, including physiological data such as electroencephalogram (EEG) and electromyogram (EMG).

[0075] The method for collecting the physiological data is as follows: multimodal physiological and behavioral tests are conducted on the assessment subjects, and the physiological data of the assessment subjects are recorded in real time.

[0076] The process of the multimodal physiological and behavioral test is as follows: the subject is fitted with an EEG sensor and an EMG sensor to record the subject's EEG and EMG signals respectively, and the EEG and EMG signals are used as the subject's physiological data.

[0077] Several behavioral tests were conducted on the subjects in sequence, and the physiological data of the subjects were recorded in real time during each behavioral test; each behavioral test was any one of the following: self-introduction test, text reading test, or gait test.

[0078] In this embodiment, such as Figure 2 As shown, the multimodal physiological and behavioral test includes: a test preparation phase, a self-introduction test, a text reading test, and a gait test. During the test preparation phase, participants wear electroencephalogram (EEG) and electromyogram (EMG) sensors to record their neural and muscular activity signals. The coordinated operation of these devices aims to comprehensively and accurately collect the participants' physiological signals during the experiment, laying the foundation for subsequent data analysis and modeling.

[0079] The self-introduction test involves the assessee introducing themselves based on a given theme.

[0080] In this implementation, the self-introduction must include: sharing your troubles, joys, and sorrows, and discussing areas where you want to improve and areas where you are satisfied.

[0081] The text reading test is as follows: the evaluation subject reads a specified text.

[0082] In this implementation, the evaluation subject needs to read aloud the designated texts "The North Wind and the Sun" and "The Wolf and the Lamb".

[0083] The gait test is as follows: the subject walks in an orderly manner within a designated area according to a predetermined route and direction.

[0084] The psychological tests were conducted using a personality scale, a General Well-Being Schedule Scale (GWB), a depression screening scale, and a depressive symptom scale. The scores of each scale were used as labels for the emotional state of the subjects.

[0085] In this implementation, personality scales are used to assess personality traits, such as the Big-5 Personality Inventory (short version), which assesses five main personality dimensions: openness, conscientiousness, extraversion, agreeableness, and neuroticism. The General Well-Being Schedule Scale (GWB) measures a user's subjective well-being. Depression screening scales, such as the PHQ-9 Depression Scale, are used to assess a user's depressive tendencies. Depressive symptom scales, such as the BDI Depression Scale, are used to further refine the quantitative analysis of depressive symptoms. Through scale evaluation, quantitative indicators reflecting an individual's depressive tendencies, personality traits, and subjective well-being can be obtained. The questionnaire scores of each scale serve as labels for the emotional state of the assessed individuals, providing supervisory information for subsequent deep learning models.

[0086] Step 2: Preprocess the physiological data of the evaluation subjects and construct a multimodal dataset, and divide the multimodal dataset into training set and test set according to the set ratio.

[0087] In this embodiment, the acquired EEG and EMG signals are standardized to improve signal quality, reduce noise and redundant information, and lay the foundation for subsequent deep learning model construction.

[0088] Step 2 further includes:

[0089] Step 2.1: Based on the timestamp information recorded in the physiological data, mark the start and end times of the assessment subject's different activity stages, and divide the physiological data into several segments according to the marked start and end times to obtain the segmented physiological data.

[0090] In this embodiment, during the acquisition of EEG and EMG signals, the assessment subject will experience different activity stages, including task execution, rest, or waiting. Within these stages, there are some activity periods unrelated to the test. Therefore, it is necessary to segment the recorded physiological data, determine the start and end times of each activity stage based on the timestamp information recorded during data acquisition, and then divide the continuous data into corresponding segments.

[0091] Step 2.2: Downsample the segmented physiological data to obtain downsampled physiological data.

[0092] In this embodiment, the acquired EEG and EMG signals typically have a high sampling rate. While a high sampling rate can retain more signal details, it significantly increases storage and processing costs. Therefore, it is necessary to downsample the EEG and EMG signals to 200Hz. This sampling rate can retain sufficient signal details while reducing the computational load for subsequent analysis and processing.

[0093] Step 2.3: Filter the downsampled physiological data to obtain filtered physiological data; a fourth-order Butterworth filter is used to bandpass filter the EEG signal in the physiological data; a bandpass filter is used to bandpass filter the EMG signal in the physiological data.

[0094] In this embodiment, the original signal often contains high-frequency noise such as environmental interference and low-frequency drift such as baseline drift, which significantly degrades signal quality. Therefore, after downsampling, the physiological data needs to be filtered. For EEG signals, a fourth-order Butterworth filter is used for bandpass filtering to filter out five frequency bands: delta (1–3 Hz), theta (4–7 Hz), alpha (8–13 Hz), beta (14–30 Hz), and gamma (31–50 Hz). For EMG signals, a 1–45 Hz bandpass filter is used to remove high-frequency noise and low-frequency interference such as baseline drift.

[0095] Step 2.4: Extract features from the filtered physiological data to obtain the time-domain and frequency-domain features of the physiological data.

[0096] In this embodiment, feature extraction includes two main categories: time-domain features and frequency-domain features. Time-domain features include the signal's mean, variance, peak-to-peak value, etc., which can reflect the basic statistical properties of the signal. Frequency-domain features are extracted through Fast Fourier Transform (FFT) to extract power spectrum, frequency band energy, etc., which can reveal the frequency distribution of the signal.

[0097] Step 2.5: For any physiological data, combine the time-domain features and frequency-domain features of the physiological data into a feature vector of the physiological data. Treat a behavioral test as an event, and use the feature vectors of all physiological data in each event as the multimodal data sequence of that event. Use the multimodal data sequences of all events to construct a multimodal dataset.

[0098] The multimodal dataset includes: an electroencephalogram (EEG) dataset and an electromyogram (EMG) dataset;

[0099] The multimodal dataset is represented as follows:

[0100]

[0101] Where X represents the multimodal dataset. If the multimodal dataset only represents the EEG dataset, then X is represented as X. EEG If the multimodal dataset only represents the electromyography (EMG) signal dataset, then X is represented as X. EMG ; This represents the multimodal data sequence of the first event E1; This represents the multimodal data sequence of the second event, E2. Represents the i-th event E i Multimodal data sequences; Represents the nth event E n The multimodal data sequence; n is the total number of events, and in this embodiment, n = 4.

[0102] In this implementation, there are four events, representing self-introduction, reading the text "The North Wind and the Sun," reading the text "The Wolf and the Lamb," and gait testing. The multimodal dataset contains physiological data from two physiological modalities: electroencephalography (EEG) and electromyography (EMG). EEG X represents physiological data under the EEG physiological modality. EMG This represents physiological data under the EMG physiological modality.

[0103] Step 2.6: Divide the multimodal dataset into training and test sets according to a preset ratio.

[0104] In this implementation, the samples are randomly divided into a training set and a test set in a 4:1 ratio. The training set is used to train the model, and the test set is used to evaluate the model's performance.

[0105] Step 3: Construct the EEG neural network model and the EMG neural network model respectively.

[0106] In this embodiment, deep learning technology is used to construct corresponding analysis models for EMG and EEG data. The construction of the neural network model mainly includes two core steps: feature representation learning and modality-specific modeling, and model training optimization and evaluation interpretation. The feature representation learning and modality-specific modeling include: using self-supervised learning methods to allow the model to learn high-quality feature representations from preprocessed data, capturing the inherent patterns and regularities of the data. Simultaneously, appropriate network structures are designed for the attributes of different modalities, such as using convolutional / attention sequence models for temporal modalities and convolutional / graph convolutional networks for spatial modalities. The network output is adjusted according to downstream tasks, and mechanisms such as multi-head attention and gated recurrent units are introduced to enhance the model's expressive power. The model training optimization and evaluation interpretation include: designing appropriate supervised training objectives, such as classification cross-entropy loss and regression mean squared error loss, and using efficient optimization algorithms such as Adam for training. Regularization strategies such as L1, L2 regularization, and dropout are introduced to prevent overfitting, and techniques such as gradient pruning and recurrent regularization are used to improve training stability. The model's generalization ability is comprehensively evaluated on the preserved dataset using metrics such as accuracy, F1 score, and root mean square error, and robustness is assessed through cross-validation. The model's internal behavior is analyzed using techniques such as gradient / attention visualization, and domain knowledge is incorporated to explain the model, thereby improving its transparency.

[0107] Both the EEG neural network model and the EMG neural network model are single-modality neural network models.

[0108] The single-modal neural network model is as follows: a convolutional neural network (CNN) model is used to extract local features from the input multimodal data sequence, wherein the CNN model consists of a single convolutional layer, represented as follows:

[0109]

[0110] in This represents the feature representation obtained by local feature extraction of the multimodal data sequence of the i-th event; Sigmoid is the activation function used to map the convolution result to the range (0,1); Conv1D represents a one-dimensional convolution operation.

[0111] In this embodiment, the input multimodal data sequence includes features from two modalities: feature vectors from electroencephalogram (EEG) signals and feature vectors from electromyography (EMG) signals. The features from both modalities are input into two independent convolutional neural network (CNN) models for local feature extraction. Both CNN models are single-layer convolutional operations with kernel sizes of 3 and strides of 1, in order to extract pattern information at different scales and levels of abstraction from the input features, ultimately obtaining a higher-level, more abstract feature representation.

[0112] The feature representation extracted by the Convolutional Neural Network (CNN) model is input into a multi-head self-attention mechanism for processing. An attention score is calculated for each physiological data point in the multimodal data sequence. This attention score is then used to adjust the value vector V. i We perform weighted summation to obtain a weighted feature representation of the physiological data.

[0113] The attention score is calculated as follows:

[0114] MultiHead(Q,K,V)=[e1,...,e r W K (3)

[0115]

[0116] Where MultiHead(Q,K,V) represents the output of the multi-head attention mechanism; Q, K, and V represent the query feature matrix, key feature matrix, and value feature matrix, respectively; e1 represents the output of the first attention head in the multi-head attention mechanism; e r e represents the output of each attention head in the r-th attention mechanism. iThis represents the output of each attention head in the i-th multi-head attention mechanism; Softmax is the activation function used to normalize the dot product similarity of all keys and queries to obtain the attention weight of each key; d is the dimension of the key vector, and This indicates scaling of the dot product result to prevent numerical instability when the dimension is too large; Q i This represents the query vector, used in conjunction with the key vector K. i Perform dot product calculation to determine attention weights; K i V is a key vector used to represent the features at each position in the input sequence and is matched with the query vector; i It is a value vector used to represent the basis for the final weighted output obtained through the attention mechanism.

[0117] In this implementation, the feature representations from the two CNN models are processed using a multi-head self-attention mechanism. This mechanism captures the correlations between features at different locations in the input sequence, thus better modeling long-term dependencies. The multi-head self-attention mechanism calculates an attention score at each time step, which is used to weight the importance of features at different locations, thereby improving the model's expressive power.

[0118] For any event, a residual connection is formed between the weighted feature representation of the physiological data in the event and the multimodal data sequence of the event. The result of the residual connection is then normalized by layer. The normalized features are then transformed nonlinearly through a feedforward network to generate the feature sequence of the event.

[0119] The feature sequence of the event is input into a bidirectional long short-term memory network (Bi-LSTM). The Bi-LSTM is used to process the forward and reverse time dependencies in the feature sequence, and the hidden state of the last time step in the feature sequence is taken as the output of the Bi-LSTM to obtain the feature vector of the event.

[0120]

[0121] in Represents the i-th event E i eigenvectors; t represents the time step; Indicates a bidirectional LSTM; h t-1 and h t+1 These represent the hidden states at time steps t-1 and t+1, respectively. Indicates event E i The feature sequence obtained after processing by a multi-head self-attention mechanism.

[0122] In this implementation, the two output sequences processed by the multi-head self-attention mechanism are fed into two Long Short-Term Memory (LSTM) networks for further processing. For each sequence input into the LSTM, the hidden state output at the last time step is taken as the final output of that LSTM, allowing the model to integrate the semantic information of the entire sequence and further enhance the expressive power of the features. LSTM, through its unique gating structure, including input gates, forget gates, and output gates, selectively retains or discards information, thus effectively handling dependencies in long-term sequences. By combining forward and backward hidden states, Bi-LSTM can capture long-term dependency information in event sequences. The event representations for different modalities are processed by a multi-head attention mechanism in the previous step, and then obtained through residual connections, layer normalization, and a feedforward network. By combining the forward and backward hidden states, Bi-LSTM can capture long-term dependency information in the event sequence.

[0123] Step 4: Fuse the EEG neural network model and the EMG neural network model to obtain a multimodal fusion model.

[0124] In this embodiment, the hidden representations of the two modal corresponding models are fused to construct a multimodal fusion model, such as... Figure 3 As shown, the main steps include: 1) Modality alignment and feature fusion: Aligning data streams from different modalities to the same time scale to address issues such as data loss and inconsistent sampling rates. Then, the feature representations of different modalities are fused. This can be done through simple concatenation or weighted summation at the feature layer, or through adaptive fusion using attention mechanisms. Furthermore, external memory and gated loop mechanisms can be used to simulate the interaction between modalities and capture modal feedback at different time steps. 2) Model fusion and joint training optimization: Fusing the outputs of single-modal models to construct a multimodal fusion model. Designing reasonable multi-task or multi-standard loss functions and employing appropriate optimization strategies to handle modal redundancy and noise. Finally, evaluating the performance of the fusion model on the retained test set, comparing it with the single-modal model using appropriate metrics, analyzing modal contribution through attention visualization, and improving model interpretability by incorporating domain knowledge.

[0125] The process of fusing the EEG neural network model and the EMG neural network model is divided into two parts: a single-modal fusion process based on the channel attention module and a multimodal fusion process based on the interactive attention mechanism.

[0126] The single-modal fusion process based on the channel attention module is as follows: A joint feature representation of multiple events is constructed using the feature vectors output by the EEG neural network model and the EMG neural network model, respectively. Global average pooling and global max pooling are then applied to the joint feature representation of the multiple events. The obtained global average pooling and global max pooling features are then processed by a two-layer fully connected network for dimensionality reduction and activation to obtain the attention weights of the events. These weights are then fused with the joint feature representation of the multiple events to generate a weighted multi-event feature representation.

[0127] For the EEG neural network model, the weighted multi-event feature representation obtained by fusing the feature vector output by the model with single-modal characteristics serves as the multi-event fusion feature H of the EEG signal. EEG For the EMG neural network model, the weighted multi-event feature representation obtained by fusing the feature vector output by the model with single-modal characteristics serves as the multi-event fusion feature H of the EMG signal. EMG .

[0128] The specific content of the single-modal fusion process based on the channel attention module is as follows:

[0129] The feature vectors of each event in the multimodal dataset are fused along the channel dimension to construct a joint feature representation of the multiple events.

[0130] The joint feature of the multiple events is represented as:

[0131]

[0132] Among them H t This represents the joint feature representation of multiple events, and The concatenation operation represents the feature representations of four events; h dim Indicates the dimension of the event feature vector; Concat represents the concatenation operation, used to fuse the feature vectors of multiple events along the channel dimension; The feature vector of the first event E1; For the i-th event E i eigenvectors; For the nth event E n eigenvectors.

[0133] The joint feature representation of multiple events is subjected to global average pooling and global max pooling respectively to obtain the global average pooling feature H. t,avg and global max pooling feature H t,max The first layer of the fully connected network is used to process the global average pooling feature H. t,avg and global max pooling feature H t,maxDimensionality reduction is performed, and then a second fully connected network is used to activate the dimensionality-reduced global average pooling features and global max pooling features respectively. The activated global average pooling features and global max pooling features are then merged to obtain the attention weights of the events.

[0134] The attention weight of the event is represented as follows:

[0135] α event =Sigmoid(W1(W0(H) t,avg ))+W1(W0(H t,max ))) (7)

[0136] Where α event The attention weights of events are used to dynamically reflect the importance of each event; Sigmoid represents the activation function; W1 is the recovery matrix; and W0 is the dimensionality reduction matrix.

[0137] Building upon this, a Channel Attention Mechanism (CAM) is introduced to optimize the event information fusion process, dynamically adjusting the contribution weight of each event in the joint representation. The joint feature representation H of multiple events... t Perform global average pooling and global max pooling respectively to obtain the feature H after average pooling. t,avg and the feature H after max pooling t,max Subsequently, the pooled feature H t,avg and H t,max Dimensionality reduction and activation are performed using a two-layer fully connected network.

[0138] The attention weight α of the event is calculated using element-wise multiplication. event Joint feature representation of multiple events H t The features are fused to obtain a weighted multi-event feature representation H. event .

[0139] The weighted multi-event feature representation H event for:

[0140] H event =α event ⊙H t (8)

[0141] Where ⊙ represents the fusion operation.

[0142] After the above processing, the fused single-modal representation can more accurately reflect the importance of information between events, providing higher quality feature representation for subsequent multimodal interaction fusion.

[0143] To capture the complex correlation between EEG and EMG signals, this implementation proposes a multimodal fusion method based on an interactive attention mechanism. This method achieves deep interaction and integration of features from the two modalities through a dual-branch cross-attention module.

[0144] The multimodal fusion process based on the interactive attention mechanism is as follows: a dual-branch cross-attention module is used to calculate the multi-event fusion features H of the EEG signals. EEG Multi-event fusion features of electromyographic signals H EMG The feature correlation between them is used to obtain the attention output from the EEG mode to the EMG mode. Attention output from EMG mode to EEG mode Then through and The data is then spliced ​​together to generate separate EEG features. and electromyographic characteristics It also integrates multimodal joint feature representations.

[0145] The dual-branch cross-attention module is as follows: one attention branch uses the multi-event fusion feature H of EEG signals. EEG For querying the multi-event fusion features H of electromyographic signals EMG For a key and a value, calculate H. EEG With H EMG The correlation of features between them, and using H EEG With H EMG Calculate the attention output from the EEG mode to the EMG mode based on the feature correlation between them. Represented as:

[0146]

[0147] Where Softmax represents the normalized exponential function; This indicates the multi-event fusion feature H of EEG signals. EEG Mapped to the Query space; This indicates the multi-event fusion feature H of EEG signals. EEG Mapped to the Key space; This indicates the multi-event fusion feature H of EEG signals. EEG Mapped to the Value space; e represents the EEG mode; m represents the EMG mode; d represents the feature dimension of Query and Key, used to prevent the attention score from being too large in the high-dimensional space, which would lead to gradient vanishing or numerical instability.

[0148] Another attention branch is based on the multi-event fusion feature H of electromyographic signals. EMGFor querying the multi-event fusion features H of EEG signals EEG For a key and a value, calculate H. EMG With H EEG The correlation of features between them, and using H EMG With H EEG Calculate the attention output from the EMG mode to the EEG mode based on the feature correlation between them. Represented as:

[0149]

[0150] in This indicates the multi-event fusion feature H of electromyographic signals. EMG Mapped to the Query space; This indicates the multi-event fusion feature H of electromyographic signals. EMG Mapped to the Key space; V t (m→e) This indicates the multi-event fusion feature H of electromyographic signals. EMG Mapped to the Value space.

[0151] In this embodiment, the multi-event fusion feature of EEG is used. Multi-event fusion features of EMG Using EEG features as input, one branch uses EEG features as the query and EMG features as the key and value; the other branch uses EMG features as the query and EEG features as the key and value.

[0152] For the first branch, the calculation of the query, key, and value is defined as follows:

[0153]

[0154] in This represents the weight matrix from the EEG modes, used to weight H... EEG Convert to Query; This represents the weight matrix from the EEG modes, used to weight H... EEG Convert to Key; This represents the weight matrix from the EEG modes, used to weight H... EEG Convert to Value.

[0155] For the second branch, the roles are swapped, defined as:

[0156]

[0157] in This represents the weight matrix from the EMG mode, used to weight H... EMG Convert to Query; This represents the weight matrix from the EMG mode, used to weight H... EMG Convert to Key; This represents the weight matrix from the EMG mode, used to weight H... EMG Convert to Value.

[0158] Subsequently, the multi-head attention outputs of each branch are concatenated to generate interactive feature representations of EEG and EMG, i.e., EEG features are generated separately. and electromyographic characteristics

[0159] The EEG characteristics Defined as:

[0160]

[0161] Concat means concatenation.

[0162] The electromyographic characteristics Defined as:

[0163]

[0164] Ultimately, the brainwave characteristics and electromyographic characteristics Further integration yields the multimodal joint feature representation H. c for:

[0165]

[0166] Through the aforementioned multimodal interaction fusion mechanism, deep integration of EEG and EMG features is achieved, significantly improving the richness and accuracy of multimodal feature representation and providing strong support for multimodal emotion state analysis.

[0167] Step 5: Build a multi-task classifier based on emotion state labels.

[0168] The method for constructing the multi-task classifier is as follows: two classification tasks are defined based on emotional state labels. One classification task is a three-level classification problem of psychological state. The three-level classification problem of psychological state is defined as a three-classification task, including three categories: normal, mild problem and severe problem. The other classification task is a two-level classification problem of five personality types. The two-level classification problem of five personality types is defined as five two-classification tasks, including five categories: openness, conscientiousness, extraversion, agreeableness and neuroticism.

[0169] A classifier is defined for each classification task to evaluate the emotional state of the physiological data corresponding to the input multimodal joint feature representation.

[0170] The classifiers for both classification tasks are defined as follows:

[0171] P i =Dropout(ReLU(W i H c +b i (16)

[0172]

[0173] Where P i W represents the hidden layer features. i 、b i 、W k 、b k These are the weight matrices used for linear transformation and the bias term, respectively; Dropout is a regularization method that prevents overfitting by randomly deactivating some neurons during training; ReLU is the activation function used to introduce non-linearity and improve the classifier's ability to express complex features; argmax represents finding the class with the highest probability from the classifier's output; b t Indicates the bias term; The evaluation results output by the classifier.

[0174] In this implementation, a three-level classification problem for psychological states is defined, with classification targets including "normal," "mild problems," and "severe problems." A two-level classification problem for five personality types is also defined, corresponding to the assessment of the Big Five personality traits: openness, conscientiousness, extraversion, agreeableness, and neuroticism.

[0175] To further improve the accuracy of emotional state assessment, this implementation proposes a multi-task framework to comprehensively analyze the psychological state and personality traits of university students using a generated multimodal fusion model. This framework utilizes a classifier of individual personality traits output by the model to account for the influence of individual differences. This framework is based on the multimodal, multi-event fusion feature representation H... cBy leveraging the auxiliary information from the personality trait labels in the preceding psychological tests, the accuracy of assessing the three-level classification of psychological states and the two-level classification of five personality types is improved. In this implementation, the process of comprehensively analyzing the psychological states and personality characteristics of college students using a multimodal fusion model mainly consists of the following two aspects: 1) The interpretation of the model evaluation results is a crucial step: First, the model's output includes indicators such as psychological states and personality characteristics. A comprehensive interpretation of the model's predictions of these indicators for the college student group is necessary to understand the overall distribution. For example, analyzing the proportion of students with poor psychological states and the distribution of different personality traits. Second, it is necessary to combine the contribution analysis of multimodal features to deeply explore the role of different modalities in the evaluation results. Through interpretability methods such as gradients, the degree of influence of each feature on the final evaluation is quantified. For example, the correlation between the quantity of negative emotional words in the text and psychological states, and the correlation between facial micro-expressions and extraversion. This helps clarify the value of multimodal data in psychological analysis and provides a basis for subsequent intervention and counseling. 2) The application of model prediction results is the ultimate goal: First, schools can provide focused attention and intervention to university students in need based on the psychological assessment results. For high-risk groups such as those with poor psychological state or paranoid personality traits, early intervention and personalized psychological counseling plans can be developed. Second, schools can optimize and improve the content and form of university students' mental health education based on the overall analysis results.

[0176] Step 6: Iteratively train the multimodal fusion model and multi-task classifier using the training set. Stop iterating when the total loss function of the multi-task classifier converges, reaches the preset number of training rounds, or meets the preset classification target, and obtain the trained multimodal fusion model and multi-task classifier.

[0177] The total loss function of the multi-task classifier is:

[0178] L total =βL m +γ(L1+L2+L3+L4+L5) (18)

[0179] Where L m β represents the cross-entropy loss of the three-level classification problem of mental states; L1, L2, L3, L4, and L5 represent the cross-entropy loss of the two-level classification problems of the five personality types; and γ represents the weight coefficients of the two-level classification problems of the five personality types.

[0180] Where L m The cross-entropy losses for L1, L2, L3, L4, and L5 are all expressed as:

[0181]

[0182] Where L represents the cross-entropy loss; N represents the total number of data in the training set; C represents the total number of classes; y lj Let be the true label of the l-th sample in the j-th category; This is the evaluation result for the l-th sample.

[0183] In this embodiment, to optimize the training of the classifier, cross-entropy is used as the loss function for the classification task, and the cross-entropy loss is defined as shown in formula (19). To comprehensively optimize the performance of psychological state and personality characteristic assessment, this embodiment proposes a weighted multi-task loss function, which sums the losses of each classification task in a weighted manner, as shown in formula (18), where β and γ are used to balance the importance of each task. The loss function is used to measure the gap between the model prediction and the true label, providing a target for model optimization. The loss is calculated when the loss converges, the preset number of training rounds is reached, or the target performance is achieved. The training process consists of forward propagation, loss calculation, backpropagation, and parameter update. The output of the trained classifier is the predicted category and probability distribution. The usage process involves inputting a new sample, obtaining the predicted type, and using the result for the actual task.

[0184] Step 7: Validate the trained multimodal fusion model and multi-task classifier using the test set to obtain the final multimodal fusion model and multi-task classifier.

[0185] Step 8: For any assessment subject, collect the subject's physiological data and input it into the final multimodal fusion model for feature fusion. Then, input the generated multimodal joint feature representation into a multi-task classifier for evaluation to obtain the assessment result of the subject's emotional state.

[0186] This embodiment provides a multi-event, multi-task-based emotion state assessment system, used to implement the aforementioned multi-event, multi-task-based emotion state assessment method, such as... Figure 4 As shown, the system includes: a data acquisition module, a data processing module, a feature fusion module, and a classification and evaluation module.

[0187] The data acquisition module is used to collect physiological data of the assessment subject and obtain emotional state labels of the assessment subject, and transmit the collected physiological data to the data processing module and the obtained emotional state labels to the classification assessment module.

[0188] The data processing module is used to perform preprocessing on the received physiological data in sequence, including segmentation, downsampling, filtering, feature extraction and splicing, and to construct a multimodal dataset using the preprocessed physiological data, and then transmit the multimodal dataset to the feature fusion module.

[0189] The feature fusion module is used to extract features from different modalities in the multimodal dataset in a single modality, and then perform single-modal fusion based on the channel attention module and multimodal fusion based on the interactive attention mechanism on the extracted feature vectors in sequence to generate a multimodal joint feature representation and transmit it to the classification evaluation module.

[0190] The classification and evaluation module is used to construct a classifier based on the emotional state label, and input the multimodal joint feature representation into the classifier to generate an emotional state evaluation result of the physiological data corresponding to the multimodal joint feature representation.

[0191] In this embodiment, an experimental evaluation of an emotion state assessment method based on multiple events and multiple tasks was conducted. The dataset was divided into a training set and a test set in a 4:1 ratio to ensure that the samples were completely independent and had no overlapping individuals, thereby avoiding the impact of data leakage on the results.

[0192] Experimental results show that the M proposed in this embodiment... 3 The ADD framework, or multi-task classifier, outperformed all baseline models in the three-level classification problem of mental states. For this problem, mental state classification was achieved from two dimensions: negative mindset and positive emotion. Specifically, the accuracy and F1 score for negative mindset classification reached 86.84% and 87.09%, respectively; while the accuracy and F1 score for positive emotion classification reached 95.12% and 95.26%, respectively. These results validate the applicability and stability of the multimodal, multi-task, and multi-event analysis framework, significantly improving the accuracy of emotion state assessment.

[0193] In terms of multi-task learning, experimental results show that adding a personality trait classification task can significantly improve model performance, with accuracy and F1 score increasing by at least 5%. This indicates that the personality classification task provides the model with additional contextual information, which helps to understand individual differences more accurately and optimize detection results.

[0194] Regarding multimodal fusion, comparative experiments show that for the three-level classification problem of mental states, the multimodal fusion method significantly outperforms the single-modal method. The joint features of EEG and EMG are consistently superior to single-modal features, further demonstrating the importance of multimodal integration in emotional state assessment.

[0195] In terms of multi-event fusion, the multi-event mechanism improved model performance compared to single-event analysis, with accuracy increasing by more than 3% and F1 score improving by more than 5%. Further analysis showed that different events contributed significantly to the model.

[0196] To verify the role of channel attention (CAM) and cross-attention (IA) mechanisms, ablation experiments were conducted. The results show that removing either CAM or IA significantly reduces model performance, with accuracy and F1 score decreasing by 3%–5%, respectively. The performance degradation is most significant when both mechanisms are removed simultaneously, indicating that both mechanisms are indispensable in feature extraction and modal interaction.

[0197] Overall, the M of this embodiment 3 The ADD framework fully leverages information from multiple modalities, tasks, and events, significantly improving classification performance on the three-level classification problem of mental states. Experimental results validate the effectiveness of the method presented in this invention, providing technical support for the accurate identification and analysis of emotional states, and possessing significant application value.

[0198] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A method for assessing emotional states based on multiple events and multiple tasks, characterized in that, The method includes the following steps: Step 1: Collect physiological data of the assessment subjects and conduct psychological tests on them. Generate emotional state labels for the assessment subjects based on the psychological test results. Step 2: Preprocess the physiological data of the evaluation subjects and construct a multimodal dataset, and divide the multimodal dataset into training set and test set according to the set ratio; Step 3: Construct neural network models for electroencephalogram (EEG) signals and electromyography (EMG) signals, respectively; Step 4: Fuse the EEG neural network model and the EMG neural network model to obtain a multimodal fusion model; The process of fusing the EEG neural network model and the EMG neural network model is divided into two parts: a single-modal fusion process based on the channel attention module and a multimodal fusion process based on the interactive attention mechanism. The single-modal fusion process based on the channel attention module is as follows: a joint feature representation of multiple events is constructed using the feature vectors output by the EEG neural network model and the EMG neural network model, respectively. Global average pooling and global max pooling are then performed on the joint feature representation of multiple events. The obtained global average pooling features and global max pooling features are then subjected to dimensionality reduction and activation processing through a two-layer fully connected network to obtain the attention weights of the events. These weights are then fused with the joint feature representation of multiple events to generate a weighted multi-event feature representation. For the EEG neural network model, the weighted multi-event feature representation obtained by fusing the feature vector output by the model with single-modal characteristics serves as the multi-event fusion feature of the EEG signal. For the EMG neural network model, the weighted multi-event feature representation obtained by fusing the feature vector output by the model with single-modal characteristics serves as the multi-event fusion feature of the EMG signal. ; The multimodal fusion process based on the interactive attention mechanism is as follows: a dual-branch cross-attention module is used to calculate the multi-event fusion features of the EEG signals. Multi-event fusion features of electromyographic signals The feature correlation between them is used to obtain the attention output from the EEG mode to the EMG mode. Attention output from EMG mode to EEG mode ; and then through and The data is spliced ​​together to generate EEG features. and electromyographic characteristics And integrates multimodal joint feature representations; Step 5: Construct a multi-task classifier based on emotion state labels; Step 6: Iteratively train the multimodal fusion model and multi-task classifier using the training set. Stop iterating when the total loss function of the multi-task classifier converges, reaches the preset number of training rounds, or meets the preset classification target, and obtain the trained multimodal fusion model and multi-task classifier. Step 7: Validate the trained multimodal fusion model and multi-task classifier using the test set to obtain the final multimodal fusion model and multi-task classifier; Step 8: For any assessment subject, collect the subject's physiological data and input it into the final multimodal fusion model for feature fusion. Then, input the generated multimodal joint feature representation into a multi-task classifier for evaluation to obtain the assessment result of the subject's emotional state.

2. The emotional state assessment method based on multi-event, multi-task as described in claim 1, characterized in that, The method for collecting physiological data in step 1 is as follows: conduct multimodal physiological and behavioral tests on the assessment subjects and record the physiological data of the assessment subjects in real time; The process of the multimodal physiological and behavioral test is as follows: the subject is fitted with an EEG sensor and an EMG sensor to record the subject's EEG signals and EMG signals respectively, and the EEG signals and EMG signals are used as the subject's physiological data. Several behavioral tests were conducted on the subjects in sequence, and their physiological data were recorded in real time during each test. Each behavioral test consisted of one of the following: a self-introduction test, a text reading test, or a gait test. The self-introduction test involves the subject introducing themselves based on a given theme; the text reading test involves the subject reading aloud a designated text; and the gait test involves the subject walking orderly within a designated area according to a predetermined route and direction. The psychological tests were conducted using a personality scale, a general well-being scale, a depression screening scale, and a depressive symptom scale. The scores of each scale were used as labels for the emotional state of the subjects. The emotional state labels include: the five personality dimensions of the assessed subject and the psychological state of the assessed subject.

3. The method for assessing emotional state based on multiple events and multiple tasks according to claim 2, characterized in that, Step 2 further includes: Step 2.1: Based on the timestamp information recorded in the physiological data, mark the start and end times of the assessment subject's different activity stages, and divide the physiological data into several segments according to the marked start and end times to obtain the segmented physiological data; Step 2.2: Downsample the segmented physiological data to obtain downsampled physiological data; Step 2.3: Filter the downsampled physiological data to obtain filtered physiological data; a fourth-order Butterworth filter is used to bandpass filter the EEG signal in the physiological data; a bandpass filter is used to bandpass filter the EMG signal in the physiological data. Step 2.4: Extract features from the filtered physiological data to obtain the time-domain and frequency-domain features of the physiological data; Step 2.5: For any physiological data, combine the time-domain features and frequency-domain features of the physiological data into a feature vector of the physiological data. Take a behavioral test as an event, and take the feature vectors of all physiological data in each event as the multimodal data sequence of that event. Use the multimodal data sequences of all events to construct a multimodal dataset; wherein the multimodal dataset includes: electroencephalogram (EEG) dataset and electromyogram (EMG) dataset; Step 2.6: Divide the multimodal dataset into training and test sets according to a preset ratio.

4. The method for assessing emotional state based on multiple events and multiple tasks according to claim 3, characterized in that, The EEG neural network model and the EMG neural network model mentioned in step 3 are both single-modality neural network models. The single-modal neural network model is: using a convolutional neural network (CNN) model to extract local features from the input multimodal data sequence, wherein the convolutional neural network (CNN) model is a single-layer convolution operation; The feature representation extracted by the Convolutional Neural Network (CNN) model is input into a multi-head self-attention mechanism for processing. An attention score is calculated for each physiological data point in the multimodal data sequence. This attention score is then used to adjust the value vector. Weighted summation is performed to obtain a weighted feature representation of the physiological data; For any event, a residual connection is formed between the weighted feature representation of the physiological data in the event and the multimodal data sequence of the event. The result of the residual connection is then normalized by layer. The normalized features are then transformed nonlinearly through a feedforward network to generate the feature sequence of the event. The feature sequence of the event is input into a bidirectional long short-term memory network (Bi-LSTM). The Bi-LSTM is used to process the forward and reverse time dependencies in the feature sequence, and the hidden state of the last time step in the feature sequence is taken as the output of the Bi-LSTM to obtain the feature vector of the event.

5. The method for assessing emotional state based on multiple events and multiple tasks according to claim 4, characterized in that, The specific content of the single-modal fusion process based on the channel attention module is as follows: The feature vectors of each event in the multimodal dataset are fused along the channel dimension to construct a joint feature representation of the multiple events; The joint feature representation of multiple events is subjected to global average pooling and global max pooling respectively to obtain the global average pooling feature. and global max pooling features The first layer of fully connected network is used to process the global average pooling features. and global max pooling features Dimensionality reduction is performed, and then a second fully connected network is used to activate the dimensionality-reduced global average pooling features and global max pooling features respectively. The activated global average pooling features and global max pooling features are then merged to obtain the attention weights of the events. The attention weight of the event is represented as follows: ； in Indicates the attention weight of an event; Indicates the activation function; To recover the matrix; It is a dimension-reduced matrix; The attention weights of events are calculated using element-wise multiplication. Joint feature representation of multiple events By fusing the data, a weighted multi-event feature representation is obtained. .

6. The method for assessing emotional state based on multiple events and multiple tasks according to claim 5, characterized in that, The dual-branch cross-attention module is as follows: one attention branch uses the multi-event fusion features of EEG signals. For querying the multi-event fusion features of electromyography signals. Calculate for key and value and The correlation of features between them, and utilize and Calculate the attention output from the EEG mode to the EMG mode based on the feature correlation between them. , is represented as: ； in This represents the normalized exponential function; This indicates the multi-event fusion characteristics of EEG signals. Mapped to the Query space; This indicates the multi-event fusion characteristics of EEG signals. Mapped to the Key space; This indicates the multi-event fusion characteristics of EEG signals. Mapped to the Value space; Indicates EEG mode; Indicates EMG mode; The feature dimension of the query and key is used to prevent the attention score from being too large in the high-dimensional space, which may lead to gradient vanishing or numerical instability. Another attention branch focuses on multi-event fusion features of electromyographic signals. For querying the multi-event fusion features of EEG signals Calculate for key and value and The correlation of features between them, and utilize and Calculate the attention output from the EMG mode to the EEG mode based on the feature correlation between them. , is represented as: ； in This indicates the multi-event fusion characteristics of electromyographic signals. Mapped to the Query space; This indicates the multi-event fusion characteristics of electromyographic signals. Mapped to the Key space; This indicates the multi-event fusion characteristics of electromyographic signals. Mapped to the Value space.

7. The method for assessing emotional state based on multiple events and multiple tasks according to claim 6, characterized in that, The method for constructing the multi-task classifier in step 5 is as follows: Define two classification tasks based on the emotional state labels, one of which is a three-level classification problem of psychological states; The three-level classification problem of psychological state is defined as a three-level classification task, including three categories: normal, mild problem and severe problem; another classification task is a two-level classification problem of five personality types; the two-level classification problem of five personality types is defined as five two-level classification tasks, including five categories: openness, conscientiousness, extraversion, agreeableness and neuroticism. A classifier is defined for each classification task to evaluate the emotional state of the physiological data corresponding to the input multimodal joint feature representation.

8. The method for assessing emotional state based on multiple events and multiple tasks according to claim 7, characterized in that, The total loss function of the multi-task classifier in step 6 for: ； in Cross-entropy loss for a three-level classification problem of mental states; Weighting coefficients representing a three-level classification problem of mental states; The cross-entropy loss represents the two-level classification problem for the five personality types, respectively. The weighting coefficients represent the two-level classification of the five personality types; in , , , , and The cross-entropy loss is expressed as: ； in Represents cross-entropy loss; This represents the total number of data points in the training set. Indicates the total number of categories; For the first The sample at the th Real labels in each category; For the first Evaluation results of one sample.

9. A multi-event, multi-task-based emotion state assessment system, used to implement the multi-event, multi-task-based emotion state assessment method of claim 1, characterized in that, The system includes: a data acquisition module, a data processing module, a feature fusion module, and a classification and evaluation module; The data acquisition module is used to collect physiological data of the assessment subject and obtain emotional state labels of the assessment subject, and transmit the collected physiological data to the data processing module and transmit the obtained emotional state labels to the classification assessment module. The data processing module is used to perform preprocessing on the received physiological data in sequence, including segmentation, downsampling, filtering, feature extraction and splicing, and to construct a multimodal dataset using the preprocessed physiological data, and then transmit the multimodal dataset to the feature fusion module. The feature fusion module is used to extract features from different modalities in the multimodal dataset in a single modality, and to perform single-modal fusion based on the channel attention module and multimodal fusion based on the interactive attention mechanism on the extracted feature vectors in sequence, so as to generate a multimodal joint feature representation and transmit it to the classification evaluation module. The classification and evaluation module is used to construct a classifier based on the emotional state label, and input the multimodal joint feature representation into the classifier to generate an emotional state evaluation result of the physiological data corresponding to the multimodal joint feature representation.

Images