Music-induced emotion recognition method based on EEG and fNIRS fusion

Abstract

This paper discloses a music-induced emotion recognition method based on the fusion of EEG and fNIRS. The method first designs a music library covering different valence-arousal dimensions. By constructing a prompt word template and innovatively combining AIGC technology, it realizes the automated generation of a large number of music clips. Then, volunteers are recruited to subjectively score the generated music clips, screening out those with good emotion-inducing effects. Based on this, the method designs a multimodal brain signal acquisition paradigm to synchronously collect EEG and fNIRS signals from subjects while listening to music. Finally, based on the model that fuses EEG and fNIRS signals, it achieves effective emotion classification. Through technological innovation and methodological integration, the method constructs a complete closed loop from music generation to brain signal analysis, providing a reproducible research paradigm and effective classification model for the field of emotion computing.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal brain signal emotion recognition, and specifically relates to a music-induced emotion recognition method based on EEG and fNIRS fusion. Background Art

[0002] Emotions have a significant impact on human health and are closely associated with a variety of mental health disorders, such as depression, anxiety, and attention deficit hyperactivity disorder. This highlights the importance of accurately identifying emotions and developing effective regulation strategies for disease prevention and treatment management. Music is widely used in daily stress reduction and mood enhancement, demonstrating its potential to promote emotion regulation. Compared with other methods (such as videos and sandplay interactions), music's low equipment requirements and cross-cultural adaptability make it a more practical and scalable tool for emotion regulation. Previous techniques have explored the potential role of music in eliciting emotions and regulating emotions, but current paradigms rely primarily on subjective reports to assess emotional responses, an approach with inherent limitations, such as introspection bias. When people attempt to understand and report their own mental states or emotions, they may be influenced by subjective factors, resulting in inaccurate or biased results. Such biases often stem from cognitive limitations in understanding their internal states or from interference from factors such as memory, emotion, and social expectations. Therefore, establishing an objective mapping between emotional states and musical patterns is crucial. This systematic approach is of great significance for advancing precise music-based emotion regulation.

[0003] Currently, methods using brain signals for emotion induction and regulation still have key limitations in terms of music stimulation, brain signal modalities, and device portability. These limitations include the following:

[0004] Limitations of Musical Stimuli: Existing technical studies mostly use small-scale music corpora, which are mainly restricted by copyright and cost. Such small-scale music corpora may lead to statistical bias and even the risk of model overfitting during the emotion recognition process. The choice of music stimuli is usually subjectively determined based on the experimenter's heuristic assumptions about emotion-music mapping, ignoring the unique emotional characteristics differences between individuals, which may reduce the robustness and universality of the model. In addition, widely circulated classical music works may inspire associations with relevant personal experiences, thereby affecting the induction of target emotions through additional cognitive processing (such as memory retrieval), rather than simply through the emotional response induced by the music. This phenomenon will interfere with the effectiveness of the experimental design.

[0005] Limitations of brain signal modalities: Most affective computing paradigms rely on single-modality neural signals, such as EEG (electroencephalography). Although EEG has temporal resolution and remains the primary modality for non-invasive affective computing, its spatial specificity is inherently limited by signal attenuation in the cerebral cortex. Multimodal fusion can alleviate this problem by providing complementary information, enabling a more comprehensive analysis of emotions. For example, fNIRS (functional Near Infrared Spectroscopy) can capture unique hemodynamic response patterns, reflecting changes in blood oxygen levels in corresponding regions.

[0006] Limitations of portability: Although music can induce emotions in a simple and direct way, the existing paradigm ignores another key factor that affects the practical application of music-induced emotions and even emotion regulation: the portability of brain signal acquisition equipment. Large and complex brain signal acquisition equipment (such as a 64-channel EEG signal acquisition cap that requires conductive gel) requires professional supporting facilities and professional operators, which increases costs and limits scalability; and in multimodal monitoring, users usually wear several different sets of equipment at the same time (such as EEG caps, eye trackers, etc.), which requires more than 30 minutes of preparation time. These operational bottlenecks fundamentally restrict the possibility of practical application. Summary of the Invention

[0007] In view of the above, the present invention provides a music-induced emotion recognition method based on the fusion of EEG and fNIRS, which collects EEG and fNIRS signals through a portable headband device to analyze the emotions induced by AIGC (Artificial Intelligence Generated Content) music.

[0008] A music-induced emotion recognition method based on EEG and fNIRS fusion includes the following steps:

[0009] (1) Design a music library based on the valence-arousal system;

[0010] (2) Using AIGC technology to automatically generate a large number of music clips for the music library;

[0011] (3) Screening music clips in the music library by scoring;

[0012] (4) Design a multimodal brain signal acquisition paradigm, play music clips from the music library, and collect EEG and fNIRS signals from different subjects under the multimodal brain signal acquisition paradigm to construct a dataset;

[0013] (5) Divide the dataset into a training set and a test set, and construct an EEG-fNIRS dual-modal classification model. Use the EEG and fNIRS signals in the training set to train the model.

[0014] (6) The EEG and fNIRS signals in the test set are input into the trained model, which can identify the emotional state corresponding to the output.

[0015] Furthermore, the music library in step (1) follows the valence-arousal circular model, in which the target emotions are distributed on a two-dimensional plane, with valence and arousal serving as the horizontal and vertical axes of the two-dimensional plane, respectively. Valence refers to the positive or negative degree of emotion, and arousal refers to the intensity of emotion. According to the model, four emotional states are set, namely HAHV (high arousal and high valence), HALV (high arousal and low valence), LAHV (low arousal and high valence), and LALV (low arousal and low valence). The music clips in the music library must be able to induce the above four emotional states. The music clips that induce HAHV have the characteristics of being exciting or loud, the music clips that induce HALV have the characteristics of being messy or sharp, the music clips that induce LAHV have the characteristics of being soothing or calm, and the music clips that induce LALV have the characteristics of being mournful, sad, or solemn.

[0016] Furthermore, in step (2), the MUSICGEN (Controllable Music Generation) model is used to automatically generate music clips. The model takes text description and melody features into consideration in the process of generating music clips. For this purpose, a set of prompt word templates needs to be designed. The templates use text sentences to describe music clips from multiple angles such as specific emotions, instrument styles, emotional tones, and situational scenes. The placeholders in the templates are used to replace various descriptive elements, thereby realizing customized generation of different types of music clips; corresponding words are filled in the prompt word templates to generate various types of prompt sentences, and these prompt sentences are input into the MUSICGEN model. The model automatically generates music clips that induce corresponding emotional states, and then labels the emotional state type corresponding to each music clip, and ensures that the number of music clips of the four emotional state types is balanced.

[0017] Furthermore, the specific implementation method of step (3) is as follows: first, music clips with obvious technical defects (such as sudden noises, long silent intervals) are removed from the music library, and then volunteers are recruited to score the music clips on the two dimensions of valence and arousal, with a score range of 1 to 9 points; for valence, scores 1 to 9 correspond to emotions from negative to positive (i.e., pleasure, the more pleasant the feeling, the higher the score); for arousal, scores 1 to 9 correspond to emotions from low arousal to high arousal (i.e., excitement, the more excited the feeling, the higher the score); for music clips of any type of emotional state, scores from 1 to 9 correspond to emotions from low arousal to high arousal (i.e., excitement, the more excited the feeling, the higher the score); N music clips closest to the ideal emotion score were selected from the test set, and a total of 4N music clips were selected, where N is a natural number greater than 1; for music clips with an emotional state type of HAHV, the ideal emotion score is 9 points for arousal and 9 points for valence; for music clips with an emotional state type of HALV, the ideal emotion score is 9 points for arousal and 1 point for valence; for music clips with an emotional state type of LAHV, the ideal emotion score is 1 point for arousal and 9 points for valence; for music clips with an emotional state type of LALV, the ideal emotion score is 1 point for arousal and 1 point for valence.

[0018] Furthermore, the specific implementation process of the multimodal brain signal acquisition paradigm in step (4) is as follows:

[0019] 4.1 The subject was seated in a shielded room and equipped with an integrated headband equipped with wireless EEG-fNIRS signal acquisition equipment to synchronously acquire EEG and fNIRS signals from the subject's forehead. The subject was then instructed to complete the following test tasks.

[0020] 4.2 Each trial begins with a 5-second rest period, during which the subject should close their eyes, adjust their mindset, and try to calm down.

[0021] 4.3 Play a piece of music from the music library for the subject twice in a row using the audio player software;

[0022] 4.4 After the music is played, the subjects are asked to rate the emotions they felt from the music clips on a scale;

[0023] 4.5 After completing the scale scoring, the subject rests for 15 seconds, and a trial is completed after the rest period.

[0024] 4.6 Repeat steps 4.2-4.5. After every five trials, the subject will complete 20 simple mental arithmetic problems. Each subject will complete a total of 40 trials.

[0025] Furthermore, the test task in step 4.1 includes two stages, each stage is divided into four blocks, each block consists of five trials, and the five trials in the same block play music clips of the same emotional state type to the subjects to minimize the influence of the potential alternation effect of induced emotions.

[0026] Furthermore, the scale score in step 4.3 is based on the SAM (Self-Assessment Manikin) scale of emotional experience, including scoring the valence, arousal and liking of the music clip, with a scoring range of 1 to 9 points; valence reflects the subject's degree of pleasure after hearing the music, the higher the pleasure, the closer the score is to 9 points, the lower the pleasure, the closer the score is to 1 point; arousal reflects the subject's degree of excitement after hearing the music, the higher the excitement, the closer the score is to 9 points, the lower the excitement, the closer the score is to 1 point; liking reflects the subject's degree of liking the music clip (exploring the subject's taste and interest preference in music), the more like it, the closer the score is to 9 points, and the more disgusted it is, the closer the score is to 1 point.

[0027] Furthermore, the EEG and fNIRS signals need to be preprocessed before being input into the model for training or prediction: for EEG signals, they are first downsampled and then band-pass filtered to limit the signal frequency range to 0.1-40 Hz; for fNIRS signals, they are first converted into optical density change signals and then band-pass filtered to limit the signal frequency range to 0.5-4 Hz to extract PPG (photoplethysmography) signals, and at the same time, the HbO (oxyhemoglobin) and HbR (deoxyhemoglobin) are calculated according to the Beer-Lambert law. The preprocessed EEG signal is combined with the PPG signal and the relative changes in HbO, HbR, and HbT concentrations as the model input. During model training, the subjects' scale scores for the music clips in the corresponding trials are used as the true value labels. The cross-entropy loss function is used to measure the error between the model's predicted probability of valence and arousal and the true value labels. The optimizer is used to iteratively update the model parameters through the gradient descent method until the loss function converges and the training is completed.

[0028] Furthermore, the EEG-fNIRS dual-modal classification model extends the EEG Conformer (a deep learning model that combines convolutional neural networks and Transformer architectures, specifically for EEG signal decoding and classification tasks) model architecture with a network branch specifically for PPG and hemoglobin concentration feature extraction, thereby fusing EEG features with PPG features and hemoglobin concentration features, and finally mapping the fused features into emotional state classification results.

[0029] Furthermore, the EEG-fNIRS dual-modal classification model first passes the input signal through a temporal convolution module, a spatial convolution module, and an average pooling layer in sequence to obtain a feature embedding representation. The temporal convolution module is used to extract the temporal features of the signal in the temporal dimension, the spatial convolution module is used to extract the channel features of the signal in the spatial dimension, and the average pooling layer is used to aggregate temporally adjacent features. The feature embedding representations of the three input signals, EEG, PPG, and hemoglobin concentration (HbO, HbR, and HbT), are then transposed and linearly projected into the same feature space and directly concatenated. They are then mapped into fixed-length sequence features through a linear transformation. The sequence features are then passed through a multi-level attention mechanism module to capture long-term global dependencies and contextual information to obtain fused features. Finally, the fused features are mapped into classification results for valence and arousal through two classifiers.

[0030] The present invention first established a basic music library covering a multidimensional emotional space (valence-arousal), ensuring the diversity and typicality of emotional stimuli through scientific dimensional division. Secondly, the present invention innovatively integrated AIGC technology to construct a prompt word template system, enabling the automated batch generation of music clips, breaking the efficiency bottleneck of traditional manual creation. Subsequently, a subjective emotion rating experiment was conducted, recruiting volunteers to perform multiple rounds of emotional ratings on the generated music clips. Based on statistical analysis, high-quality clips with significant emotion-inducing effects were screened. Furthermore, the present invention designed a standardized multimodal brain signal acquisition paradigm, synchronously collecting EEG and fNIRS signals while subjects listened to music, constructing a comprehensive dataset containing time series features and changes in brain hemodynamics. Finally, the present invention developed a multimodal signal fusion algorithm to construct an emotion classification model that integrates EEG time-frequency features and fNIRS brain activation patterns. Cross-validation demonstrated high-precision classification of multidimensional emotional states. In summary, through technological innovation and methodological integration, the present invention has established a complete closed loop from music generation to brain signal analysis, providing a reproducible research paradigm and effective classification model for the field of emotion computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the overall process of the music-induced emotion recognition method based on the fusion of EEG and fNIRS in the present invention.

[0032] Figure 2 This is a prompt word template and prompt example diagram for the AIGC of the present invention to generate a music clip.

[0033] Figure 3 Schematic diagram of the process of EEG and fNIRS signal acquisition experimental phase in the present invention.

[0034] Figure 4Schematic diagram of the structure of the EEG-fNIRS dual-modal classification model in the present invention. DETAILED DESCRIPTION

[0035] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] like Figure 1 As shown, the music-induced emotion recognition method based on the fusion of EEG and fNIRS of the present invention has the following specific steps:

[0037] (1) A music library was designed in which each music clip follows the valence-arousal cycloid model of American psychologist James A. Russell. In this model, emotions are distributed on a two-dimensional plane, with valence and arousal as the horizontal and vertical axes respectively. The valence dimension refers to the positive or negative degree of emotion, ranging from positive emotions (such as happiness and satisfaction) to negative states (such as sadness and anger), while the arousal dimension refers to the intensity of emotion, ranging from low activation states (such as calmness and relaxation) to high activation states (such as excitement and tension). We assume that each music clip in the music library can evoke a specific emotional state. Therefore, all music clips can evoke four different emotional states: high arousal / high valence (HAHV), high arousal / low valence (HALV), low arousal / high valence (LAHV), and low arousal / low valence (LALV). Music that induces HAHV emotions is characterized by being passionate and loud; music that induces HALV emotions is characterized by being chaotic and sharp; music that induces LAHV emotions is characterized by being soothing and calm; and music that induces LALV emotions is characterized by being mournful, poignant, or solemn.

[0038] (2) In order to evoke emotions through music clips, the present invention attempts to use AIGC technology to automatically generate music clips. AIGC technology can generate content based on input keywords or requirements (prompt words). This technology can automatically generate music clips on a large scale. This embodiment adopts the MUSICGEN model, which is a controllable music generation model based on autoregression and Transformer. It takes into account both text description and melody features when generating music clips. In short, by inputting a prompt sentence, MUSICGEN can generate the required music clip. The input prompt is crucial for the music clip generated by MUSICGEN. Therefore, this embodiment designs a prompt word template, such as Figure 2 As shown, the generated music clips can effectively evoke emotions. The template describes the music clips from the perspective of specific emotions, instrument styles, and situational features or scenes through text sentences. The placeholders in the template can be replaced with various descriptive elements, thereby realizing customized generation of different types of music clips.

[0039] Specifically, valence adjectives can be words that describe the positive or negative nature of an emotion (e.g., positive emotions: happy, joyful; negative emotions: sad, melancholy), while arousal adjectives are used to characterize the intensity or activation level of an emotion (e.g., energized, calm). Instrumental or style elements specify the instrument type or musical genre (e.g., guitar, drums, piano, strings). Emotional tone conveys the overall mood or feeling (e.g., exciting, soothing, peaceful), and context or scene provides an appropriate background for the music (e.g., party, quiet night, festival). The prompt sentences generated using this template cover the four emotional states of HAHV, HALV, LAHV, and LALV. Ultimately, by inputting various types of prompt sentences into MUSICGEN, hundreds of music clips were generated, each initially labeled according to the emotional state conveyed by the input prompt sentence. This implementation generated a total of 236 music clips, with a balanced number of music clips corresponding to each emotional state.

[0040] (3) A manual screening method is used to ensure that the emotion type induced by the music clip is consistent with the label of the music clip. This embodiment first excludes music clips with technical defects (for example, sudden noises, long silent intervals, etc.), and then recruits volunteers to score the generated music clips. Volunteers need to score each music clip on two dimensions: valence and arousal, with a score range of 1 to 9. Valence indicates whether the music is pleasant or unpleasant. The higher the pleasantness, the closer the score is to 9 points, and the lower the pleasantness, the closer the score is to 1 point. Arousal indicates whether the music is exciting or not. The higher the excitement, the closer the score is to 9 points, and the lower the excitement, the closer the score is to 1 point. Based on the scores of the volunteers, this embodiment selects 100 music clips (25 music clips of each of the four types). The overall goal of selecting music clips is to select music with a score closest to the ideal emotion score. At the same time, the number of music clips to be selected should be greater than 100. For the four emotional states of HAHV, HALV, LAHV, and LALV, the ideal emotional scores were arousal 9 / valence 9, arousal 9 / valence 1, arousal 1 / valence 9, and arousal 1 / valence 1, respectively.

[0041] (4) A multimodal brain signal acquisition paradigm was designed to simultaneously acquire EEG signals and fNIRS signals, such as Figure 3 As shown, the experiment consisted of two phases, each phase was divided into four blocks, each block consisted of five trials, and each trial lasted about 90 seconds.

[0042] This implementation requires the subject to sit in a shielded room and follow instructions to complete a series of tasks. During this time, they wear an integrated EEG-fNIRS signal wireless acquisition headband device provided by Rongnao Technology to collect EEG-fNIRS signals from the subject's frontal area. The EEG signal collects two channels (FP1 and FP2) of scalp EEG in the frontal lobe area, and the fNIRS signal collects changes in near-infrared light intensity from eight channels (8 laser light sources and 8 avalanche photodiode detectors) in the frontal lobe area, thereby obtaining changes in blood oxygen content in the frontal area.

[0043] Before each trial begins, there will be a 5-second preparation period, during which the subjects are asked to close their eyes and adjust their mental state. Afterwards, the music will be played for 60 seconds, specifically, a 30-second music clip will be played twice in a row; during the music playing, the subjects need to keep their eyes closed, their bodies still, and avoid moving their heads or eyes in order to better focus on the music and feel the emotional changes in it; after the music ends, the subjects need to rate the valence, arousal, and liking of the music. It should be noted that liking is different from valence. The former reflects more of an individual's interest or preference in music rather than an emotional response. For example, some people may like music that makes them feel sad. After each complete trial, the participants have a 15-second break.

[0044] To minimize the potential impact of alternating emotions, the same emotional music was used across all five trials within a single block. Additionally, between blocks, participants completed simple math problems to help them focus and mitigate any residual effects of the emotions evoked in the previous block. Under this paradigm, each participant completed the two-stage experimental task over approximately 80 minutes.

[0045] (5) The collected raw EEG signals are standardized and preprocessed to generate normalized time series data suitable for deep neural network input, and the collected raw fNIRS signals are calculated to obtain the hemoglobin concentration.

[0046] This implementation first downsamples the raw EEG data to 100 Hz, then uses a bandpass filter to limit the signal frequency range to 0.1-40 Hz. To ensure that each EEG signal segment is aligned with the emotional label of its corresponding music clip, this implementation intercepts the signal from 25 to 55 seconds after the onset of stimulation as the usable time period, considering this stage to be the critical period for emotional response. Finally, baseline correction is performed on each intercepted time period.

[0047] In this embodiment, the raw fNIRS signal is converted into an optical density change signal and a 0.5-4 Hz bandpass filter is applied to extract the photoplethysmography signal, which reflects the heart rate change.

[0048] The absorption properties of light (Beer-Lambert law) can be used to establish a relationship between near-infrared light attenuation and hemoglobin concentration. The scattering properties of light can be used to enable optical observation of nearby brain tissue on the scalp surface. This embodiment uses a modified Beer-Lambert law to calculate hemoglobin content, specifically as follows:

[0049]

[0050] in: OD is the light attenuation, C is the concentration of the light-absorbing substance, ε is the extinction coefficient, L is the straight-line distance between the emitting photoelectrode and the receiving photoelectrode, DPF is the path length correction factor, which represents the multiple of the path length increased due to the scattering effect; DPF and L Multiply by the corrected actual optical path length, G It represents the sum of the light intensity attenuation caused by other factors in addition to the light intensity attenuation caused by HbO and HbR. DPF It is difficult to accurately determine the parameters, as different tissues of subjects of different genders and ages have different DPF In practical applications, the empirical values ​​obtained in previous studies are often used, and the parameters G It is almost unknowable, but studies usually focus not on the absolute concentration of hemoglobin, but on the relative concentration, which is calculated as follows:

[0051]

[0052] The incident light intensity of the fNIRS device was kept constant (wavelength λ ), select a time point before the experimental stimulus appears t 0 as the baseline, calculate the t After time t The relative change of the output light intensity at time 1 can be used to list the equations between the relative attenuation of two different wavelengths of near-infrared light and the relative change of the two hemoglobin concentrations:

[0053]

[0054]

[0055] Solving the problem can give the relative concentration changes of HbO and HbR at any time:

[0056]

[0057] After obtaining the relative changes in the concentrations of HbO and HbR, the relative change in HbT concentration can be obtained by summing the two.

[0058] For the classification task, this embodiment designs and implements a new deep neural network architecture, namely the EEG-fNIRS dual-modal classification model, which extends the Conformer model with a branch specifically for hemoglobin content signals to achieve the fusion of multimodal signals. Figure 4 As shown in the figure, the input of the EEG-fNIRS dual-modality classification model in this embodiment is three types of preprocessed physiological signals: EEG, blood oxygenation (HbO / HbR / HbT), and PPG. All three modal data are represented and learned using a unified feature embedding process. Specifically, a convolutional neural network is first used to extract temporal features in the temporal dimension to capture the dynamic changes of the signal within a local time range. Subsequently, a convolutional network is applied in the spatial dimension to extract cross-channel interaction information, enhancing the spatial structure modeling capability. Finally, the temporal features are compressed and aggregated through an average pooling operation to obtain a compact representation vector for each modality.

[0059] In the feature fusion stage, the feature representations of the three modalities are first transposed and linearly projected to unify their dimensional structure and semantic space, and then they are spliced ​​in the feature dimension to form an integrated multimodal feature sequence, which is further mapped into a fixed-length sequence representation through linear transformation and serves as the input of the attention module.

[0060] The attention module is based on the self-attention mechanism, which aims to model the global dependencies and contextual semantic connections between multimodal features. Q ,key K Sum V The weighted similarity between the three realizes dynamic weighting of key feature dimensions, effectively improving the model's sensitivity to information relevance and important areas.

[0061]

[0062] Among them: Attention represents the self-attention mechanism, d K is the dimension of the key, Softmax represents the normalized exponential function, and MLP represents the multilayer perceptron.

[0063] In order to enhance the representation ability of the model and capture deeper feature interactions, the model is stacked N The attention module enables each layer to gradually deepen the global modeling based on the features of the previous layer, thereby effectively improving the model's ability to recognize complex spatiotemporal patterns.

[0064] Finally, the fused features are mapped into classification results on valence and arousal through two classifiers, and the classifier is composed of two fully connected layers FC connection.

[0065] This implementation uses the cross-entropy loss function during model training to measure the difference between the model's predicted probability distribution and the true label:

[0066]

[0067] in: N is the sample size, y ic Indicates the i samples in the category c The true label on (one-hot encoding), Represents the probability of the model prediction; the optimizer uses AdamW (Adam optimizer with weight decay), which combines an adaptive learning rate mechanism with a weight decay strategy to help improve the stability and generalization performance of model training. This implementation uses inter-subject and intra-subject classification strategies to verify the rationality of the framework. For inter-subject classification, leave-one-subject cross-validation is used; for intra-subject classification, k-fold cross-validation is used.

[0068] Using the aforementioned multimodal brain signals, including EEG, fNIRS-derived hemodynamic signals (HbO, HbR, HbT), and PPG, we evaluated classification performance under six modality combinations: EEG only, PPG only, HbO / HbR / HbT only, EEG and PPG fusion, EEG and HbO / HbR / HbT fusion, and EEG, PPG, and HbO / HbR / HbT tri-modality fusion. In Table 1, Hb is used to represent HbO / HbR / HbT signals. By comparing the performance of different combinations, we aim to explore the contribution of each physiological signal and its complementarity in the emotion recognition task.

[0069] Table 1

[0070]

[0071] As can be seen from Table 1, in most tasks, the combination of all modalities performed best; compared with a single modality, multimodal integration significantly improved the emotion classification performance for valence and arousal. Notably, under the accuracy (ACC) metric, PPG performed most prominently in valence recognition, while EEG performed best in arousal classification. At the same time, PPG performed better in arousal classification than in valence, with an accuracy rate approximately 3% higher. This may be because an individual's heart rate (measured by PPG) fluctuates more significantly when listening to faster-paced music, making it easier to reflect changes in arousal. Across all tasks, arousal classification performance consistently outperformed valence, whether using ACC or MF1 (average F1 score); this phenomenon suggests that existing physiological signals are more reliable and discriminable in detecting arousal than valence.

[0072] The above description of the examples is intended to facilitate understanding and application of the present invention by those skilled in the art. It will be apparent that those skilled in the art can readily make various modifications to the above-described embodiments and apply the general principles described herein to other embodiments without requiring inventive effort. Therefore, the present invention is not limited to the above-described embodiments, and improvements and modifications made by those skilled in the art based on the disclosure of the present invention should fall within the scope of protection of the present invention.

Claims

1. A music-induced emotion recognition method based on EEG and fNIRS fusion, comprising the following steps: (1) Design a music library based on the valence-arousal system; (2) Using AIGC technology to automatically generate a large number of music clips for the music library; (3) Screening music clips in the music library by scoring; (4) Designing a multimodal brain signal acquisition paradigm, playing music clips from the music library and collecting EEG and fNIRS signals from different subjects under the multimodal brain signal acquisition paradigm to construct a dataset; (5) Divide the dataset into training and test sets, and construct an EEG-fNIRS dual-modal classification model. The model is trained using the EEG and fNIRS signals in the training set. The EEG-fNIRS dual-modal classification model extends the EEG Conformer model architecture with a network branch specifically for PPG and hemoglobin concentration feature extraction, thereby fusing EEG features with PPG features and hemoglobin concentration features, and finally mapping the fused features into emotional state classification results; Regarding fNIRS signals, they were first converted into optical density change signals, and then a bandpass filter was used to limit the signal frequency range to 0.5-4 Hz to extract the PPG signal. At the same time, the relative changes in HbO, HbR, and HbT concentrations were calculated according to the Beer-Lambert law. (6) The EEG and fNIRS signals in the test set are input into the trained model, which can identify the emotional state corresponding to the output.

2. The method for music-induced emotion recognition based on EEG and fNIRS fusion according to claim 1, characterized in that: The music library in step (1) follows the valence-arousal circular model, in which the target emotions are distributed on a two-dimensional plane, with valence and arousal serving as the horizontal and vertical axes of the two-dimensional plane, respectively. Valence refers to the positive or negative degree of emotion, and arousal refers to the intensity of emotion. According to the model, four emotional states are set, namely HAHV, HALV, LAHV and LALV. The music clips in the music library must be able to induce the above four emotional states. The music clips that induce HAHV have the characteristics of being exciting or loud, the music clips that induce HALV have the characteristics of being messy or sharp, the music clips that induce LAHV have the characteristics of being soothing or calm, and the music clips that induce LALV have the characteristics of being mournful, sad or solemn.

3. The music-induced emotion recognition method based on EEG and fNIRS fusion according to claim 2, characterized in that: In the step (2), the MUSICGEN model is used to automatically generate music clips. The model takes text descriptions and melody features into consideration during the generation of music clips. For this purpose, a set of prompt word templates needs to be designed. The templates use text sentences to describe music clips from multiple perspectives such as specific emotions, instrument styles, emotional tones, and situational scenes. The placeholders in the templates are used to replace various descriptive elements, thereby realizing customized generation of different types of music clips. Corresponding words are filled in the prompt word templates to generate various types of prompt sentences. These prompt sentences are input into the MUSICGEN model. The model automatically generates music clips that induce corresponding emotional states, and then labels the emotional state type corresponding to each music clip, and ensures that the number of music clips of the four emotional state types is balanced.

4. The method for music-induced emotion recognition based on EEG and fNIRS fusion according to claim 2, characterized in that: The specific implementation method of step (3) is as follows: first, music clips with obvious technical defects are eliminated from the music library, and then volunteers are recruited to score the music clips in terms of valence and arousal, with the score range being 1 to 9 points; for valence, scores of 1 to 9 correspond to emotions from negative to positive; for arousal, scores of 1 to 9 correspond to emotions from low arousal to high arousal; for music clips of any type of emotional state, N music clips closest to the ideal emotional score are selected, and a total of 4N music clips are selected, where N is a natural number greater than 1; for music clips with an emotional state type of HAHV, the ideal emotional score is 9 points for arousal and 9 points for valence; for music clips with an emotional state type of HALV, the ideal emotional score is 9 points for arousal and 1 point for valence; for music clips with an emotional state type of LAHV, the ideal emotional score is 1 point for arousal and 9 points for valence; and for music clips with an emotional state type of LALV, the ideal emotional score is 1 point for arousal and 1 point for valence.

5. The method for music-induced emotion recognition based on EEG and fNIRS fusion according to claim 2, characterized in that: The specific implementation process of the multimodal brain signal acquisition paradigm in step (4) is as follows: 4.1 The subject was seated in a shielded room and equipped with an integrated headband equipped with wireless EEG-fNIRS signal acquisition equipment to synchronously acquire EEG and fNIRS signals from the subject's forehead. The subject was then instructed to complete the following test tasks. 4.2 Each trial begins with a 5-second rest period, during which the subject should close their eyes, adjust their mindset, and try to calm down. 4.3 Play a piece of music from the music library for the subject twice in a row using the audio player software; 4.4 After the music is played, the subjects are asked to rate the emotions they felt from the music clips on a scale; 4.5 After completing the scale scoring, the subject rests for 15 seconds, and a trial is completed after the rest period; 4.6 Repeat steps 4.2 to 4.

5. After every five trials, the subject is required to complete 20 simple mental arithmetic problems. Each subject is required to complete a total of 40 trials.

6. The method for music-induced emotion recognition based on EEG and fNIRS fusion according to claim 5, characterized in that: The test task in step 4.1 includes two stages, each stage is divided into four blocks, each block consists of five trials, and the five trials in the same block play music clips of the same emotional state type to the subjects.

7. The method for music-induced emotion recognition based on EEG and fNIRS fusion according to claim 5, characterized in that: The scale scoring in step 4.3 is based on the SAM scale of emotional experience, which includes scoring the valence, arousal, and liking of the music clips on a scale of 1 to 9. Valence reflects the degree of pleasure the subject feels after hearing the music. The higher the pleasure, the closer the score is to 9, and the lower the pleasure, the closer the score is to 1. The arousal level reflects the subject's excitement level after hearing the music. The higher the excitement level, the closer the score is to 9 points, and the lower the excitement level, the closer the score is to 1 point. The liking level reflects the subject's liking for the music clip. The more they like it, the closer the score is to 9 points, and the more they dislike it, the closer the score is to 1 point.

8. The method for music-induced emotion recognition based on EEG and fNIRS fusion according to claim 1, characterized in that: The EEG and fNIRS signals are preprocessed before being input into the model for training or prediction: the EEG signal is first downsampled and then bandpass filtered to limit the signal frequency range to 0.1-40 Hz; the fNIRS signal is first converted into an optical density change signal and then bandpass filtered to limit the signal frequency range to 0.5-4 Hz to extract the PPG signal. The relative changes in HbO, HbR, and HbT concentrations are calculated according to the Beer-Lambert law. The preprocessed EEG signal is then combined with the PPG signal and the relative changes in HbO, HbR, and HbT concentrations as the input to the model. During model training, the scale ratings of the subjects for the music clips in the corresponding trials are used as the true value labels. The cross-entropy loss function is then used to measure the error between the model's predicted probability of valence and arousal and the true value labels. The model parameters are continuously iteratively updated using the optimizer using the gradient descent method until the loss function converges and training is complete.

9. The method for music-induced emotion recognition based on EEG and fNIRS fusion according to claim 1, characterized in that: The EEG-fNIRS dual-modal classification model first passes the input signal through a temporal convolution module, a spatial convolution module, and an average pooling layer in sequence to obtain a feature embedding representation. The temporal convolution module is used to extract the temporal features of the signal in the temporal dimension, the spatial convolution module is used to extract the channel features of the signal in the spatial dimension, and the average pooling layer is used to aggregate temporally adjacent features. The feature embedding representations of the three input signals, EEG, PPG, and hemoglobin concentration, are then directly concatenated after being transposed and linearly projected into the same feature space and mapped into fixed-length sequence features through a linear transformation. The sequence features are then passed through a multi-level attention mechanism module to capture long-term global dependencies and contextual information to obtain fused features. Finally, two classifiers are used to map the fused features into classification results for valence and arousal.

Images