Facial electromyogram emotion recognition method based on muscle synergy attention and multi-instance learning

Abstract

The application discloses a facial electromyogram emotion recognition method based on muscle synergistic attention and multi-instance learning, and comprises the following steps: obtaining facial electromyogram signals, and obtaining emotion induction segments through pretreatment and window segmentation; extracting single-channel features of muscle channels according to short-time windows, and constructing cross-channel synergistic features according to the signal relationship among the muscle channels; calculating the attention weights of the muscle channels through a muscle synergistic attention mechanism, fusing the single-channel features to obtain window-level fusion features; combining the window-level fusion features with the cross-channel synergistic features to generate window-level embedding features; taking the emotion induction segments as sample bags and the window-level embedding features corresponding to the short-time windows as sample instances, and obtaining segment-level representations through multi-instance attention aggregation; and outputting emotion recognition results according to the segment-level representations. The application improves the accuracy of facial electromyogram emotion recognition.

Description

Technical Field

[0001] This invention belongs to the field of physiological signal processing, and in particular relates to a facial electromyography emotion recognition method based on muscle-coordinated attention and multi-instance learning. Background Technology

[0002] Facial electromyography (EMG) signals can objectively reflect subtle facial muscle activities induced by emotions and are unaffected by factors such as lighting conditions and facial occlusion. Therefore, emotion recognition based on facial EMG signals has received widespread attention in the fields of affective computing and human-computer interaction. Currently, mainstream facial EMG emotion recognition methods typically segment the original signal into short-time windows and extract time-domain and frequency-domain features such as mean absolute amplitude, root mean square value, and autoregressive coefficients from each window. Then, classifiers such as support vector machines, random forests, or neural networks are used for emotion discrimination. In recent years, with the development of deep learning technology, some studies have begun to introduce attention mechanisms and multi-instance learning frameworks to automatically mine key time segments and channel information in physiological signal analysis. Furthermore, existing research has shown that different facial muscles do not operate independently during emotional expression; there are specific co-activation patterns among the corrugator supercilii, zygomaticus major, and depressor anguli oris muscles. This co-activation relationship has potential value in distinguishing negative emotions such as fear and sadness.

[0003] However, existing methods still have the following shortcomings: First, traditional methods typically input the features of each muscle channel into the classifier after simple concatenation, without explicitly modeling the collaborative relationships between different facial muscles. This makes it difficult to fully utilize the interactive information of the corrugator supercilii, zygomaticus major, and depressor anguli oris muscles in emotional expression, thus limiting the ability to distinguish emotions. Second, in experimental scenarios where audiovisual stimuli induce emotions, each longer emotion-inducing signal is only labeled with an overall segment-level label, while the intensity of emotional expression in each short-term window within the segment is not uniform, and not all windows stably represent the emotion. If segment-level labels are directly assigned to each short-term window for window-level supervised training, label noise will be introduced, interfering with the model's effective learning of key emotional response windows. Third, although existing methods partially introduce channel attention mechanisms, they often design channel attention and temporal attention separately, failing to simultaneously utilize single-channel muscle activity information, cross-channel muscle collaboration information, and the selection information of key time windows within a unified framework. This results in the model's stability and interpretability in emotion recognition under limited channel conditions still needing improvement. Summary of the Invention

[0004] To overcome the problems and shortcomings of existing technologies, this invention proposes a facial electromyography (EMG) emotion recognition method based on muscle-coordinated attention and multi-instance learning. This invention uses three-channel facial EMG signals as input, simultaneously extracting single-channel multi-domain features and cross-channel muscle-coordinated features within a short time window. It adaptively weights the contributions of the corrugator supercilii, zygomaticus major, and depressor anguli oris muscles using a three-channel muscle-coordinated attention module. Furthermore, each emotion-inducing signal segment is used as a sample packet, and its internal short-time window is used as a sample instance. Fragment-level representations are obtained through multi-instance attention aggregation, ultimately outputting the probabilities of three emotion categories: fear, sadness, and neutral.

[0005] To achieve the above objectives, this invention provides a facial electromyography emotion recognition method based on muscle-coordinated attention and multi-instance learning, comprising: Facial electromyography (EMG) signals are acquired, and the EMG signals are preprocessed and segmented into windows to obtain emotion-inducing fragments composed of short-term windows. Based on the short time window, single-channel features of muscle channels are extracted, and cross-channel collaborative features are constructed based on the signal relationships between muscle channels. Based on the single-channel features, the attention weights of the muscle channels are calculated through the muscle collaborative attention mechanism, and the single-channel features are fused based on the attention weights to obtain window-level fused features. Based on the window-level fusion features and the cross-channel collaborative features, a window-level embedding feature is generated; The emotion-inducing fragments are used as sample packages, and the window-level embedding features corresponding to the short-term windows in the sample packages are used as sample instances. All sample instances are weighted and aggregated through a multi-instance attention aggregation mechanism to obtain fragment-level representations. Based on the fragment-level representation, the emotion recognition result is output.

[0006] Preferably, the process of obtaining the emotion-inducing segment composed of a short time window includes: Acquire three-channel facial electromyography signals, namely the corrugator supercilii channel, the zygomaticus major channel, and the depressor anguli oris channel; The three-channel facial electromyography signals are filtered, notched, and normalized, and then segmented non-overlapping according to a preset window length to obtain a short-time window. Each emotion-inducing signal is defined as a sample packet containing multiple short-time windows.

[0007] Preferably, the process of extracting single-channel features of the muscle channel includes: For each muscle channel in each short time window, multi-domain features are extracted. These multi-domain features include mean absolute amplitude, root mean square value, waveform length, variance, multi-order autoregressive coefficients, and window energy, forming a single-channel feature vector for the corresponding muscle channel.

[0008] Preferably, the process of constructing cross-channel synergistic features based on the signal relationships between muscle channels includes: Multiple muscle channels are combined in pairs to form muscle pairs; Calculate the synergy index between the muscle pairs, which includes logarithmic energy ratio, normalized energy difference, and rectified envelope correlation coefficient; The synergistic indices of the muscle pairs are concatenated to form a cross-channel synergistic feature vector.

[0009] Preferably, the process of calculating the synergy index between the muscle pairs includes: The logarithmic energy ratio and the normalized energy difference are calculated based on the window energy of the two muscle channels in the muscle pair. The correlation coefficient of the rectified envelope is calculated based on the rectified envelope of the electromyographic data of the two muscle channels in the muscle pair.

[0010] Preferably, the process of obtaining window-level fusion features includes: The single-channel feature vectors of each muscle channel within each short time window are input into a network with shared parameters to obtain the attention score of each muscle channel. The attention scores of each muscle channel are normalized to obtain the attention weights of each muscle channel. By using the attention weights of each muscle channel, the corresponding single-channel feature vectors are weighted and summed to obtain window-level fusion features.

[0011] Preferably, the process of generating window-level embedding features includes: The window-level fusion feature is concatenated with the cross-channel collaborative feature to obtain a fusion feature vector; The fused feature vector is input into a fully connected layer for mapping to obtain window-level embedding features of a preset dimension.

[0012] Preferably, the process of obtaining fragment-level representations includes: Each segment of the emotion-inducing signal is defined as a sample bag, and the window-level embedding features corresponding to each short-term window within the emotion-inducing signal are defined as sample instances. Calculate the attention score for each sample instance, and normalize the attention scores within the same sample bag to obtain the temporal attention weight for each sample instance; The window-level embedding features of each sample instance are weighted and summed using the temporal attention weights to obtain a fragment-level representation.

[0013] Preferably, the process of outputting emotion recognition results includes: The fragment-level representation is input into the classification layer, and the probability of the emotion-inducing fragment belonging to each emotion category is output. The emotion category with the highest probability is selected as the emotion recognition result of the emotion-inducing segment.

[0014] Preferably, the method further includes a training process, the training process comprising: Using the segment-level emotion label of each emotion-inducing signal as the supervision label, the cross-entropy loss function is used to jointly train the muscle-coordinated attention mechanism, the window-level embedding generation process, the multi-instance attention aggregation mechanism, and the classification output process.

[0015] Compared with the prior art, the present invention has the following advantages and technical effects: First, by constructing cross-channel collaborative features, this invention explicitly calculates the logarithmic energy ratio, normalized energy difference, and rectified envelope correlation coefficient among three muscle pairs—corrugator supercilii, zygomaticus major, and depressor anguli oris—effectively characterizing the relative enhancement, relative inhibition, and synchronous changes among muscles under different emotional states. This overcomes the defect of lost collaborative information caused by the simple splicing of traditional single-channel features, thereby significantly improving the ability to distinguish negative emotions such as fear and sadness.

[0016] Second, the present invention adopts a segment-level multi-instance learning mechanism, which uses the entire emotion-inducing signal as a sample package, the internal short-term window as a sample instance, and uses segment-level labels as supervision signals for aggregation training. This avoids the label noise problem caused by forcibly assigning segment-level labels to each short-term window, enabling the model to automatically identify and focus on the key time window within the segment that contributes more to emotion discrimination.

[0017] Third, the three-channel muscle-coordinated attention module constructed in this invention works in collaboration with the multi-instance temporal attention mechanism within the same framework. It can provide muscle channel contribution weights and temporal window attention distributions while outputting emotion recognition results, enabling the model to have both high recognition accuracy and clear biological interpretability under the limited sensing conditions of only three-channel facial electromyography signals. Attached Figure Description

[0018] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention. Detailed Implementation

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0021] like Figure 1 As shown, this embodiment provides a facial electromyography emotion recognition method based on muscle-coordinated attention and multi-instance learning, including: Facial electromyography (EMG) signals are acquired, preprocessed, and segmented into windows to obtain emotion-inducing fragments composed of short-term windows. Based on the short time window, single-channel features of muscle channels are extracted, and cross-channel collaborative features are constructed based on the signal relationships between muscle channels; Based on the single-channel features, the attention weights of the muscle channels are calculated through the muscle collaborative attention mechanism, and the single-channel features are fused based on the attention weights to obtain window-level fused features. Window-level embedding features are generated based on window-level fusion features and cross-channel collaborative features; The emotion-inducing segments are used as sample packages, and the window-level embedding features corresponding to the short-term windows in the sample packages are used as sample instances. All sample instances are weighted and aggregated through a multi-instance attention aggregation mechanism to obtain segment-level representations. Based on the fragment-level representation, the emotion recognition result is output.

[0022] This embodiment first filters, normalizes, and segments the original three-channel facial electromyography (EMG) signals to obtain emotion-inducing segments composed of multiple short-term windows. Then, for each short-term window, multi-domain EMG features are extracted from the corrugator supercilii, zygomaticus major, and depressor anguli oris channels, and cross-channel synergistic features between the three muscle pairs are calculated. Next, a three-channel muscle synergistic attention module is constructed to calculate the contribution weights of the three muscle channels in each window, forming window-level muscle attention features. Finally, each emotion-inducing segment is used as a sample package, and its internal short-term windows are used as sample instances. A multi-instance attention aggregation module is used to obtain segment-level representations and output the probabilities of three emotion categories: fear, sadness, and neutral. This embodiment can provide a clear and interpretable technical solution for emotion recognition based on facial EMG signals, utilizing single-channel muscle activity, cross-channel muscle synergistic relationships, and key time window information within segments, all within the constraints of only three-channel facial EMG signals and segment-level emotion labels.

[0023] Furthermore, the process of obtaining the emotion-inducing fragment constituted by the short-term window includes: Three-channel facial electromyography signals were acquired, namely the corrugator supercilii channel, the zygomaticus major channel, and the depressor anguli oris channel. The three-channel facial electromyography signals were filtered, notched, and normalized, and then segmented non-overlapping according to a preset window length to obtain a short time window. Each emotion-inducing signal is defined as a sample packet containing multiple short-time windows.

[0024] Furthermore, this embodiment acquires three-channel facial electromyography (EMG) signals from any subject, with the three channels being the corrugator supercilii channel, the zygomaticus major channel, and the depressor anguli oris channel; the three-channel facial EMG signals are subjected to 20-500Hz bandpass filtering, 50Hz notch filtering, and z-score normalization, and are segmented non-overlapping according to a 0.5-second window length; 300 short-time windows are obtained from each 150-second emotion-inducing signal segment; Specifically, this embodiment acquires and preprocesses three-channel facial electromyography signals, including: Three-channel facial electromyography (EMG) signals were acquired from any subject. The three channels originated from the corrugator supercilii, zygomaticus major, and depressor anguli oris muscles, and were designated as Channel 1, Channel 2, and Channel 3, respectively. The emotional label for each emotion-inducing signal was fear, sadness, or neutral.

[0025] The original three-channel facial electromyography (EMG) signals were subjected to bandpass filtering (20-500Hz) and notch filtering (50Hz), followed by z-score normalization for each channel. Each 150-second signal segment was divided into non-overlapping segments with a 0.5-second window length. At a sampling rate of 1926Hz, each short-time window contained L=963 sampling points, and each 150-second signal segment contained T=300 short-time windows.

[0026] Let the sample packet corresponding to the i-th segment of the 150-second emotion-inducing signal be . : (1) in, Let represent the t-th short-time window in the i-th signal segment, and: (2) In the formula, This represents the electromyographic data of the c-th facial muscle channel within the t-th short-time window of the i-th signal segment.

[0027] Furthermore, the process of extracting single-channel features of muscle channels includes: For each muscle channel in each short time window, multi-domain features are extracted. These multi-domain features include mean absolute amplitude, root mean square value, waveform length, variance, multi-order autoregressive coefficients, and window energy, forming a single-channel feature vector for the corresponding muscle channel.

[0028] Furthermore, in this embodiment, for the t-th short-time window in the i-th segment of the emotion-inducing signal, the mean absolute amplitude (MAV), root mean square value (RMS), waveform length (WL), variance (VAR), third-order autoregressive coefficients (AR1, AR2, AR3) and window energy (E) are extracted from the three channels respectively to form an 8-dimensional single-channel feature vector for each channel. The single-channel feature vector of the c-th channel in the t-th short-time window is: Specifically, the process of extracting single-window, single-channel, multi-domain features in this embodiment includes: For the t-th short-time window and the c-th channel in the i-th signal segment, from Eight fixed features are extracted: mean absolute amplitude (MAV), root mean square value (RMS), waveform length (WL), variance (VAR), third-order autoregressive coefficients (AR1, AR2, AR3), and window energy (E). For simplicity, the subscript i is omitted below.

[0029] The mean absolute value (MAV) is calculated as follows: (3) The root mean square (RMS) value is calculated as follows: (4) The waveform length WL is calculated as follows: (5) The variance VAR is calculated as follows: (6) The window energy E is calculated as follows: (7) The third-order autoregressive coefficients are determined by equation (8): (8) This yields the single-channel feature vector of the c-th channel in the t-th short-time window. : (9) In this embodiment, each short-time window contains three channels, thus each short-time window generates three 8-dimensional single-channel feature vectors. , and .

[0030] Furthermore, the process of constructing cross-channel synergistic features based on the signal relationships between muscle channels includes: Multiple muscle channels are combined in pairs to form muscle pairs; Calculate the synergy indices between muscle pairs, including logarithmic energy ratio, normalized energy difference, and rectified envelope correlation coefficient; The synergistic indices of muscle pairs are concatenated to form a cross-channel synergistic feature vector.

[0031] Furthermore, in this embodiment, the cross-channel muscle synergy feature vector is composed of the logarithmic energy ratio, normalized energy difference, and rectified envelope correlation coefficient of three muscle pairs. The three muscle pairs are, in order, corrugator supercilii-zygomaticus major, corrugator supercilii-depressor anguli oris, and zygomaticus major-depressor anguli oris.

[0032] Specifically, the process of constructing cross-channel muscle synergy features in this embodiment includes: Within the t-th short-time window, three muscle pairs are constructed: corrugator supercilii-zygomaticus major, corrugator supercilii-depressor anguli oris, and zygomaticus major-depressor anguli oris. For uniform representation, let muscle pairs (a, b) belong to set P: (10) For any muscle pair (a, b), first, based on the window energy in step 2... and Calculate the logarithmic energy ratio : (11) Where ε is a constant to prevent the denominator from being zero; in this embodiment, ε is taken as 10. -6 .

[0033] Secondly, calculate the normalized energy difference. : (12) Next, calculate the correlation coefficient of the rectified envelope. .Will and Take the absolute values ​​to obtain the rectified envelope. and The correlation coefficient is: (13) By sequentially concatenating the three synergy indices of the three muscle pairs, a 9-dimensional cross-channel muscle synergy feature vector for the t-th short-time window is obtained. : (14) in, This represents the logarithmic energy ratio of the corrugator supercilii to the zygomaticus major muscle. This represents the logarithmic energy ratio of the corrugator supercilii to the depressor anguli oris muscles. This represents the logarithmic energy ratio of the zygomaticus major muscle to the depressor anguli oris muscle; the subscripts d and ρ have the same meaning as r.

[0034] Furthermore, the process of calculating synergy indices between muscle pairs includes: Calculate the logarithmic energy ratio and normalized energy difference based on the window energy of the two muscle channels in the muscle pair; The correlation coefficient of the rectified envelope is calculated based on the rectified envelope of the electromyographic data of the two muscle channels in the muscle pair.

[0035] Furthermore, in this embodiment, the logarithmic energy ratio, normalized energy difference, and rectified envelope correlation coefficient are calculated for three muscle pairs: corrugator supercilii-zygomaticus major, corrugator supercilii-depressor anguli oris, and zygomaticus major-depressor anguli oris. The calculation results of the three muscle pairs are then concatenated into a 9-dimensional cross-channel muscle collaborative feature vector. For any muscle pair (a, b), the logarithmic energy ratio, normalized energy difference, and rectified envelope correlation coefficient are respectively: Where ε is a constant to prevent the denominator from being zero.

[0036] Furthermore, the process of obtaining window-level fusion features includes: The single-channel feature vectors of each muscle channel within each short time window are input into a network with shared parameters to obtain the attention score of each muscle channel. The attention scores of each muscle channel are normalized to obtain the attention weights of each muscle channel. By using the attention weights of each muscle channel, the corresponding single-channel feature vectors are weighted and summed to obtain window-level fusion features.

[0037] Furthermore, in this embodiment, the single-channel feature vectors of the three channels within each short-term window are input into a single-layer perceptron with shared parameters to obtain the attention scores of the three channels, and the three-channel muscle attention weights are obtained by Softmax normalization; the single-channel feature vectors of the three channels are weighted and summed using the three-channel muscle attention weights to obtain window-level muscle attention features. The calculation method for the three-channel muscle attention weights is as follows: in, This represents the attention weight of the c-th muscle channel in the t-th short window. This represents window-level muscle attention features.

[0038] Specifically, this embodiment constructs a three-channel muscle-coordinated attention module including: The three single-channel feature vectors of the t-th short-time window , and Each single-layer perceptron with shared parameters is input separately, and attention scores for the three muscle channels are calculated. : (15) in, , q are the parameters obtained during training. The three attention scores are normalized using the Softmax function to obtain the three-channel muscle attention weights. : (16) in, This represents the contribution weight of the c-th muscle channel to emotion recognition within the t-th short-time window. This attention weight is then used to perform a weighted summation of the three single-channel feature vectors to obtain the window-level muscle attention features. : (17) Because each It is an 8-dimensional vector, therefore It is also an 8-dimensional vector.

[0039] Furthermore, the process of generating window-level embedding features includes: The window-level fusion features are concatenated with the cross-channel collaborative features to obtain the fusion feature vector; The fused feature vector is input into the fully connected layer for mapping to obtain window-level embedded features of a preset dimension.

[0040] Furthermore, in this embodiment, window-level muscle attention features are concatenated with cross-channel muscle collaboration feature vectors and input into a fully connected layer to obtain 32-dimensional window-level embedding features; The calculation method for window-level embedding features is as follows: in, This is a cross-channel muscle coordination feature vector. It is a 32-dimensional window-level embedding feature.

[0041] Specifically, the process of generating window-level embedding features in this embodiment includes: The obtained window-level muscle attention features Synergistic Feature Vectors with Cross-Channel Muscles By concatenating the features, we obtain the fused features of the t-th short-time window. : (18) In this embodiment, It is 8-dimensional. It is 9-dimensional, therefore It is 17-dimensional. The input is a fully connected layer and passed through a ReLU activation function to obtain 32-dimensional window-level embedded features. : (19) Furthermore, the process of obtaining fragment-level representations includes: Each segment of the emotion-inducing signal is defined as a sample bag, and the window-level embedding features corresponding to each short-term window within the emotion-inducing signal are defined as sample instances. Calculate the attention score for each sample instance, and normalize the attention scores within the same sample bag to obtain the temporal attention weight for each sample instance; By using temporal attention weights to perform weighted summation of the window-level embedding features of each sample instance, a fragment-level representation is obtained.

[0042] Furthermore, in this embodiment, each 150-second emotion-inducing signal segment is defined as a sample packet, and the 300 window-level embedding features within the signal segment are defined as sample instances. The temporal attention weight of each sample instance is calculated using a multi-instance attention mechanism, and the 300 window-level embedding features are weighted and summed to obtain a segment-level representation. The calculation method for the multi-instance attention mechanism is as follows: in, Let H represent the temporal attention weight of the t-th short-term window for fragment-level emotion recognition, and let H represent the fragment-level representation.

[0043] Specifically, the fragment-level multi-instance attention aggregation process involved in this embodiment includes: Each 150-second segment of emotion-inducing signal is defined as a sample packet. The window-level embedding features corresponding to 300 short-time windows within this signal segment are used. , ... Defined as a sample instance. For each sample instance, calculate its multi-instance attention score. : (20) in, , `v` are the parameters obtained during training. Softmax normalization is applied to 300 attention scores within the same segment to obtain the temporal attention weights. : (twenty one) By using temporal attention weights to perform a weighted summation of 300 window-level embedding features, a segment-level representation of the i-th segment of the emotion-inducing signal is obtained. : (twenty two) in, This represents the contribution weight of the t-th short-time window in the i-th signal segment to segment-level emotion recognition.

[0044] Furthermore, the process of outputting emotion recognition results includes: Input the fragment-level representation into the classification layer and output the probability that the emotion-inducing fragment belongs to each type of emotion; The emotion category with the highest probability is selected as the emotion recognition result of the emotion-inducing segment.

[0045] Furthermore, in this embodiment, the fragment-level representation is input into the Softmax classification layer, and the output emotion-inducing signal is the probability of belonging to three emotion categories: fear, sadness, and neutral. The category corresponding to the highest probability is selected as the emotion recognition result.

[0046] The Softmax classification layer outputs a three-dimensional class vector: The three dimensions correspond to fear, sadness, and neutral emotions, respectively, and the predicted category is the emotion category corresponding to the highest probability value in the class vector.

[0047] Specifically, the fragment-level emotion classification and model training process involved in this embodiment includes: Fragment-level representation Inputting the softmax classification layer yields three types of emotion probability vectors. : (twenty three) in, , and Let represent the probabilities of the i-th signal segment belonging to fear, sadness, and neutral emotions, respectively. The category with the highest probability is ultimately selected as the predicted emotion category for the i-th signal segment. (twenty four) Furthermore, the method also includes a training process, which includes: Using the segment-level emotion label of each emotion-inducing signal as the supervision label, the cross-entropy loss function is used to jointly train the muscle-coordinated attention mechanism, the window-level embedding generation process, the multi-instance attention aggregation mechanism, and the classification output process.

[0048] Furthermore, in this embodiment, during training, segment-level emotion labels of 150-second emotion-inducing signals are used as supervision labels, without setting separate short-term window labels. The cross-entropy loss function is used to train the three-channel muscle-coordinated attention module, window-level embedding layer, multi-instance attention aggregation module, and Softmax classification layer. (25) In this embodiment, the optimizer is Adam, the learning rate is set to 0.001, the batch size is set to 9 sample bags, and the maximum number of training rounds is set to 100 rounds. After training, any 150-second three-channel facial electromyography signal to be identified is input, and the above steps are followed sequentially to obtain the probabilities of three emotion categories: fear, sadness, and neutrality, and the final predicted category.

[0049] This embodiment validates the method using three-channel facial electromyography (EMG) data from 27 subjects. Each subject had three 150-second emotion-inducing signals for each of the three emotion categories: fear, sadness, and neutral, resulting in a total of 243 fragment-level sample packets. The experiment employed a user-relevant three-fold cross-validation approach: in each fold, two signals for each emotion category from each subject were selected as training samples, and the remaining signal was used as the test sample. The test segments were rotated until all three signals had been tested once. To reduce fluctuations caused by random initialization of the model, each experiment was repeated five times, and the mean and standard deviation of each test result were used as the evaluation results. Evaluation metrics included fragment-level recognition accuracy, precision, recall, and F1 score.

[0050] To verify the effectiveness of the method in this embodiment, under the same preprocessing, training set, and test set partitioning conditions, the method in this embodiment is compared with Support Vector Machine (SVM), Random Forest (RF), 3rd-order Autoregressive Features with Cascaded Forest (AR-DF), Multilayer Perceptron (MLP) based on fixed features, and Attention Multi-Instance Learning (ABMIL) without muscle-assisted attention. Each method uses a 150-second emotion-inducing segment as the final evaluation unit. The segment-level emotion recognition results of different methods are shown in Table 1.

[0051] Table 1 As shown in Table 1, the average recognition accuracy of the method in this embodiment is 83.21%, which is higher than that of SVM, RF, AR-DF, MLP, and ABMIL. Specifically, compared to ABMIL without muscle-coordinated attention, the recognition accuracy of the method in this embodiment is improved by 7.28 percentage points; and compared to the AR-DF method, it is improved by 5.47 percentage points. These results indicate that combining cross-channel muscle coordination, muscle channel attention, and fragment-level multi-instance aggregation can improve the fragment-level discrimination ability of the three emotion categories.

[0052] To analyze the contribution of each component module to the recognition performance, ablation experiments were conducted under the same data partitioning. Using single-channel multi-domain features and average pooling as the basic model, cross-channel muscle coordination features, a three-channel muscle coordination attention module, and a multi-instance temporal attention aggregation module were added sequentially. The ablation experiment results for each component module are shown in Table 2.

[0053] Table 2 As shown in Table 2, the recognition accuracy improved by 4.06 percentage points after adding cross-channel muscle coordination features, indicating that the relative energy and synchronous change relationship between the three muscle pairs can provide supplementary information in addition to single-channel features. After further adding muscle coordination attention, the recognition accuracy continued to improve by 2.73 percentage points. After replacing average pooling with multi-instance temporal attention, the complete model reached 83.21%, which is a cumulative improvement of 11.63 percentage points compared with the basic model, indicating that there are differences in the contribution of different short windows within the segment to the overall emotion label.

[0054] This embodiment explicitly constructs cross-channel muscle synergy features. By calculating the energy ratio, normalized energy difference, and rectified envelope correlation coefficient of three muscle pairs—corrugator supercilii, zygomaticus major, and depressor anguli oris—this embodiment can describe the relative enhancement, relative inhibition, and synchronous changes of facial muscles under different emotional states, avoiding the loss of muscle synergy information caused by relying solely on single-channel features.

[0055] This embodiment constructs a three-channel muscle-coordinated attention module, which can adaptively weight the contributions of the three facial muscle channels within each short time window, enabling the model to simultaneously utilize the activity features of individual muscles and the importance differences of different muscle channels when performing emotion recognition.

[0056] This embodiment employs a segment-level multi-instance learning mechanism, which uses each 150-second emotion-inducing signal as a sample packet and 300 0.5-second short windows as sample instances. It automatically aggregates key windows through time attention weights, avoiding label noise caused by directly equating segment-level emotion labels with all window labels.

[0057] This embodiment has interpretable output. The model can output three-channel muscle attention weights and window time attention weights, thereby indicating the muscle channels that contribute significantly to emotion classification and the key emotional response time windows.

[0058] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A facial electromyographic emotion recognition method based on muscle-coordinated attention and multi-instance learning, characterized in that, include: Facial electromyography (EMG) signals are acquired, and the EMG signals are preprocessed and segmented into windows to obtain emotion-inducing fragments composed of short-term windows. Based on the short time window, single-channel features of muscle channels are extracted, and cross-channel collaborative features are constructed based on the signal relationships between muscle channels. Based on the single-channel features, the attention weights of the muscle channels are calculated through the muscle collaborative attention mechanism, and the single-channel features are fused based on the attention weights to obtain window-level fused features. Based on the window-level fusion feature and the cross-channel collaborative feature, a window-level embedding feature is generated; The emotion-inducing fragments are used as sample packages, and the window-level embedding features corresponding to the short-term windows in the sample packages are used as sample instances. All sample instances are weighted and aggregated through a multi-instance attention aggregation mechanism to obtain fragment-level representations. Based on the fragment-level representation, the emotion recognition result is output.

2. The method according to claim 1, characterized in that, The process of obtaining an emotion-inducing fragment consisting of a short window includes: Acquire three-channel facial electromyography signals, namely the corrugator supercilii channel, the zygomaticus major channel, and the depressor anguli oris channel; The three-channel facial electromyography signals are filtered, notched, and normalized, and then segmented non-overlapping according to a preset window length to obtain a short-time window. Each emotion-inducing signal is defined as a sample packet containing multiple short-time windows.

3. The method according to claim 1, characterized in that, The process of extracting single-channel features of muscle channels includes: For each muscle channel in each short time window, multi-domain features are extracted. These multi-domain features include mean absolute amplitude, root mean square value, waveform length, variance, multi-order autoregressive coefficients, and window energy, forming a single-channel feature vector for the corresponding muscle channel.

4. The method according to claim 1, characterized in that, The process of constructing cross-channel synergistic features based on the signal relationships between muscle channels includes: Multiple muscle channels are combined in pairs to form muscle pairs; Calculate the synergy index between the muscle pairs, which includes logarithmic energy ratio, normalized energy difference, and rectified envelope correlation coefficient; The synergistic indices of the muscle pairs are concatenated to form a cross-channel synergistic feature vector.

5. The method according to claim 4, characterized in that, The process of calculating the synergy index between the muscle pairs includes: The logarithmic energy ratio and the normalized energy difference are calculated based on the window energy of the two muscle channels in the muscle pair. The correlation coefficient of the rectified envelope is calculated based on the rectified envelope of the electromyographic data of the two muscle channels in the muscle pair.

6. The method according to claim 1, characterized in that, The process of obtaining window-level fusion features includes: The single-channel feature vectors of each muscle channel within each short time window are input into a network with shared parameters to obtain the attention score of each muscle channel. The attention scores of each muscle channel are normalized to obtain the attention weights of each muscle channel. By using the attention weights of each muscle channel, the corresponding single-channel feature vectors are weighted and summed to obtain window-level fused features.

7. The method according to claim 1, characterized in that, The process of generating window-level embedding features includes: The window-level fusion feature is concatenated with the cross-channel collaborative feature to obtain a fusion feature vector; The fused feature vector is input into a fully connected layer for mapping to obtain window-level embedding features of a preset dimension.

8. The method according to claim 1, characterized in that, The process of obtaining fragment-level representations includes: Each segment of the emotion-inducing signal is defined as a sample bag, and the window-level embedding features corresponding to each short-term window within the emotion-inducing signal are defined as sample instances. Calculate the attention score for each sample instance, and normalize the attention scores within the same sample bag to obtain the temporal attention weight for each sample instance; The window-level embedding features of each sample instance are weighted and summed using the temporal attention weights to obtain a fragment-level representation.

9. The method according to claim 1, characterized in that, The process of outputting emotion recognition results includes: The fragment-level representation is input into the classification layer, and the probability of the emotion-inducing fragment belonging to each emotion category is output. The emotion category with the highest probability is selected as the emotion recognition result of the emotion-inducing segment.

10. The method according to claim 1, characterized in that, The method further includes a training process, which includes: Using the segment-level emotion label of each emotion-inducing signal as the supervision label, the cross-entropy loss function is used to jointly train the muscle-coordinated attention mechanism, the window-level embedding generation process, the multi-instance attention aggregation mechanism, and the classification output process.

Images