A music enjoyment degree recognition method based on music-electroencephalogram feature fusion
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- LANZHOU UNIV
- Filing Date
- 2025-07-08
- Publication Date
- 2026-07-03
Smart Images
Figure CN120837101B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of music signal recognition technology, and in particular to a method for recognizing the degree of music enjoyment based on music-EEG feature fusion. Background Technology
[0002] Mental health disorders are prevalent in the general population, becoming a major global public health issue. Globally, 71% of people with mental health disorders lack access to necessary mental health services, particularly in low-income countries. Music therapy is a safe, inexpensive, and promising non-pharmacological intervention, proven to have positive effects on mental disorders including depression, anxiety, and stress. However, traditional music therapy relies heavily on therapist assessment and experience, lacking objective assessment indicators and standardized, universally applicable intervention protocols. Furthermore, the development of objective physiological signals and artificial intelligence offers significant opportunities for music therapy.
[0003] Music therapy is a treatment method that uses music to promote physical and mental health, specifically by utilizing the engaging power of music to activate neural plasticity, evoke emotions, and alter cognitive function. These changes can be tracked using electroencephalography (EEG), which provides real-time, objective data on emotional states, allowing for personalized interventions to optimize treatment outcomes. EEG characteristics during music therapy show significant changes; for example, specific musical stimuli can enhance EEG frequencies associated with emotional processing and cognitive engagement, indicating a musically aroused emotional response. Therefore, analyzing EEG features that characterize these neural responses significantly enhances the calculability of participants' emotions during music therapy. Recent research has shown that EEG waveforms can effectively predict a participant's level of enjoyment with music, and this enjoyment has a significant impact on the relief of depression. This study also indicates that individuals with higher levels of musical enjoyment exhibit greater brain-music synchronization, manifested as higher coherence between music and EEG waveforms. However, researchers believe that due to the complexity of natural polyphonic music, the amplitude envelope of music and neural signals may not be the sole or primary characteristic of neural synchronization. Compared to temporal features such as envelope, spectral information improves the detectability of neural synchronization in natural music because it can capture features such as tempo, familiarity, and beat salience.
[0004] Current tools for studying neural synchronization in natural music include Reliability Component Analysis (RCA) and Time Response Function (TRF). RCA extracts neural components that respond stably to musical stimuli, reducing interference from noise and irrelevant brain activity. TRF uses a linear regression model to fit the time dependency between musical features and EEG signals, directly revealing how musical information is encoded in the brain.
[0005] However, the drawback of the prior art is that the explanation of possible physiological mechanisms in the prior art makes it impossible to determine whether the observed neural synchronization reflects a phase-locked unidirectional coupling between the stimulus rhythm and the activity of the neural oscillator, or a convolution of the stimulus and the neural activity induced by the stimulus, which further leads to poor accuracy and robustness in the identification of the degree of music enjoyment. Summary of the Invention
[0006] The purpose of this invention is to provide a method for recognizing the level of music enjoyment based on music-EEG feature fusion, thereby solving the problems of poor accuracy and robustness in the existing technology for recognizing the level of music enjoyment.
[0007] To achieve the above objectives, this invention provides a method for recognizing the degree of music enjoyment based on music-EEG feature fusion, comprising the following steps:
[0008] S1. Extract the Mel frequency cepstral coefficients of the music signal, preprocess the music signal and perform fast Fourier transform to obtain the spectral feature values of the music signal, and then perform filtering, dimensionality reduction and decorrelation processing on the spectral feature values of the music signal in sequence to obtain the music feature coefficients.
[0009] S2. Extract the Mel frequency cepstral coefficients of the EEG signal, optimize and update the order of the Mel frequency cepstral coefficients of the EEG signal, so that the comprehensive error value between the EEG feature reconstruction signal of the EEG signal Mel frequency cepstral coefficients and the original EEG signal meets the preset conditions.
[0010] S3. The FastDTW algorithm is used to align the music feature coefficients and the reconstructed EEG feature signals in the feature dimension and time axis to obtain the alignment result;
[0011] S4. Identify the degree of music enjoyment of the person being identified based on the alignment results.
[0012] In some embodiments of this application, S1, preprocessing and performing a fast Fourier transform on the music signal includes:
[0013] S11. Perform pre-emphasis processing on the music signal to obtain a signal with enhanced high-frequency components;
[0014] S12. The pre-emphasized signal is sequentially framed and windowed to reduce spectrum leakage;
[0015] S13. Perform a Fast Fourier Transform on the windowed value to obtain the frequency domain information of the music signal, expressed as:
[0016]
[0017]
[0018] Where s[m,n] is the window value, S[m,k] is the Fast Fourier Transform value of the window value, n is the window length limit, N is the window length, m is the frame index, k is the transform parameter, and N FFT For the window length of the Fast Fourier Transform, P music [m,k] represents the frequency domain information of the music signal.
[0019] In some embodiments of this application, in S1, the spectral feature values of the music signal are sequentially filtered, reduced in dimension, and decorrelated to obtain music feature coefficients, including:
[0020] S14. Construct a Mel filter based on the inverse conversion result of the conversion relationship between the Mel frequency and the preceding frequency. Use the Mel filter to filter the frequency domain information obtained in S13. The expression is:
[0021]
[0022] Where i is the Mel range parameter, f[i-1], f[i], and f[i+1] are all linear frequency response functions, and H[i,k] is the filter value;
[0023]
[0024] Where M[m,i] is the Mel filter spectrum;
[0025] S15. The cosine transform method is used to reduce the dimensionality of the Mel filter spectrum. The expression is:
[0026] E[m,i]=log(M[m,i]);
[0027] Where E[m,i] is the dimensionality reduction result;
[0028] S16. The DCT algorithm is used to eliminate the correlation between Mel spectra and extract spectral features to obtain the music feature coefficients, the expression of which is:
[0029]
[0030] Among them, MFCC music [m,o] represents the musical characteristic coefficients, and o is a set of cepstral coefficients.
[0031] In some embodiments of this application, in S2, extracting the Mel frequency cepstral coefficients of the EEG signal includes:
[0032] S21. Extract the Mel-frequency cepstral coefficients of the EEG signal, the expression of which is:
[0033]
[0034] Among them, MFCC EEG [m,o] represents the extracted Mel frequency cepstral coefficients, and PEEG[m,k] represents the frequency domain information of the EEG signal.
[0035] In some embodiments of this application, in S2, optimizing and updating the order of the Mel frequency cepstral coefficients of the EEG signal so that the combined error value between the reconstructed EEG feature signal of the Mel frequency cepstral coefficients and the original EEG signal meets preset conditions includes:
[0036] S22. Perform inverse DCT, exponential operation, inverse Mel filtering, inverse FFT and overlap summation on the Mel frequency cepstral coefficients of the EEG signal in sequence to obtain the EEG feature reconstruction signal;
[0037] S23. The combined error value is obtained based on the time-domain error and the frequency-domain error, expressed as follows:
[0038]
[0039] in, For time-domain error, For frequency domain error, The total error value is given by T, where T is the time domain range and K is the frequency domain range. EEG The raw EEG signal, X reconstructed PSD(X) is a signal reconstructed from EEG features. EEG ) represents the power spectral density of the original EEG signal, PSD(X) reconstructed ) represents the power spectral density of the signal reconstructed from EEG features, w time For time-domain error weights, w freq For frequency domain error weights;
[0040] S24. Iteratively update the order of the reconstructed signal based on the comprehensive error value; if the reconstruction error of the current order is lower than the preset error, update the order and reconstruction error; if the reconstruction error of the current order is higher than the preset error, reduce the search step size and iterate again.
[0041] In some embodiments of this application, in S3, the FastDTW algorithm is used to align the music feature coefficients and the reconstructed EEG feature signals in the feature dimension and time axis, and the alignment results include:
[0042] S31. Obtain the similarity value between the audio signal and the EEG signal that satisfy the preset constraints, wherein the preset constraints include at least boundary constraints, continuity constraints, and monotonicity constraints.
[0043] The expression for similarity value is:
[0044]
[0045] Where dist is the similarity value, w c Here, C represents the number of constraints.
[0046] S32. For the music feature coefficients and EEG feature reconstructed signals, a downsampling algorithm is used to generate a multi-scale representation. The sequence length is halved each time it is generated, and multi-resolution decomposition is performed. Standard DTW is performed on the low resolution to calculate the initial low-resolution path.
[0047] S33. The initial low-resolution path is mapped to the high-resolution space through the projection function, and the original resolution path is obtained by optimizing it step by step based on the resolution level.
[0048] S34. A radius constraint algorithm is used for local adjustment to obtain the optimal path at the original resolution. The FastDTW algorithm is then used for cumulative calculation to obtain the distance matrix. Based on this distance matrix, the alignment path is calculated, expressed as:
[0049] path'={(i1,j1),…(i Z ,j Z )};
[0050] Where path' is the alignment path, (i1,j1) is the element value in the first row and first column of the distance matrix, (i Z ,j Z ) represents the element value in the Z-th row and Z-th column of the distance matrix.
[0051] In some embodiments of this application, in step S4, identifying the degree of music enjoyment of the person to be identified based on the alignment result includes:
[0052] The current music signal and the EEG signal of the person to be identified are acquired. Based on steps S1-S3, the current music signal and the EEG signal of the person to be identified are processed to obtain the current signal alignment path. The degree of music enjoyment of the person to be identified is identified based on the current signal alignment path.
[0053] The advantages and beneficial effects of this invention compared to the prior art are:
[0054] 1. This invention proposes an adaptive search strategy to optimize the order of MFCCs in EEG signals. This strategy can dynamically optimize the extraction dimension of MFCC order based on the specific characteristics of the signal. In particular, compared with existing fixed-order methods, the adaptive dynamic search strategy balances the accuracy and redundancy of MFCC order, thereby improving the effectiveness and robustness of signal feature extraction.
[0055] 2. This invention proposes a novel music-to-brainwave (MEFF) fusion mechanism based on FastDTW to analyze the synchronicity of music-to-brainwave signals for use in tasks involving the identification of participants' level of musical enjoyment. This mechanism uses the FastDTW algorithm to solve the problem of efficient alignment of the MFCCs feature dimensions of the two signals. Compared to results using only EEG features, fusing music and EEG features significantly improves recognition accuracy.
[0056] 3. This invention improves the interpretability of results by analyzing the recognition performance of brain regions and the correlation between the proposed features and the music enjoyment rating. This work provides a practical application reference for improving mental health interventions through personalized music therapy, and also offers suggestions for lead optimization, potentially promoting the widespread adoption of wearable EEG sound intervention systems.
[0057] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0058] Figure 1 This is a schematic diagram illustrating the steps of a method for recognizing the degree of music enjoyment based on music-EEG feature fusion in an embodiment of the present invention;
[0059] Figure 2 This is a flowchart of the Mel-Cepstral Coefficients extraction process for music signals according to an embodiment of the present invention.
[0060] Figure 3 This is a flowchart of the Mel-Cepstral Coefficient Extraction Process for EEG Signals in an embodiment of the present invention.
[0061] Figure 4 This is a schematic diagram of the optimal path calculation for dynamic time planning in an embodiment of the present invention;
[0062] Figure 5 This is a flowchart of the constraint search region according to an embodiment of the present invention;
[0063] Figure 6 This is a flowchart of the backtracking process for finding the optimal path according to an embodiment of the present invention;
[0064] Figure 7 This is a schematic diagram of brain region division according to an embodiment of the present invention;
[0065] Figure 8This is a diagram showing the order distribution of the Mel-Cepstral coefficients of the EEG signal in an embodiment of the present invention. Detailed Implementation
[0066] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product is in use. They are used only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," and "connect" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.
[0067] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
[0068] like Figure 1 As shown, this invention provides a method for recognizing the degree of music enjoyment based on music-EEG feature fusion, comprising the following steps:
[0069] S1. Extract the Mel-frequency cepstral coefficients of the music signal, preprocess the music signal and perform a Fast Fourier Transform, and then sequentially filter, reduce dimensions, and decorrelate the spectral features of the music signal to obtain the music feature coefficients. The specific process is as follows: Figure 2 As shown.
[0070] S2. Extract the Mel frequency cepstral coefficients of the EEG signal, and optimize and update the order of the Mel frequency cepstral coefficients of the EEG signal so that the comprehensive error value between the EEG feature reconstruction signal of the EEG signal Mel frequency cepstral coefficients and the original EEG signal meets the preset conditions.
[0071] S3. The FastDTW algorithm is used to align the music feature coefficients and EEG feature reconstruction signals on the feature dimension and time axis to obtain the alignment result.
[0072] S4. Identify the degree of music enjoyment of the person being identified based on the alignment results.
[0073] In some embodiments of this application, S1, preprocessing and performing a fast Fourier transform on the music signal includes:
[0074] S11. Perform pre-emphasis processing on the music signal to obtain the Mel frequency cepstral coefficients after the high-frequency components are enhanced;
[0075] S12. The pre-emphasized signal is sequentially framed and windowed to reduce spectrum leakage;
[0076] S13. Perform a Fast Fourier Transform on the windowed value to obtain the frequency domain information of the music signal, expressed as:
[0077]
[0078] Where s[m,n] is the window value, S[m,k] is the Fast Fourier Transform value of the window value, n is the window length limit, N is the window length, m is the frame index, k is the transform parameter, and N FFT For the window length of the Fast Fourier Transform, P music [m,k] represents the frequency domain information of the music signal.
[0079] In some embodiments of this application, in S1, the spectral feature values of the music signal are sequentially filtered, reduced in dimension, and decorrelated to obtain music feature coefficients, including:
[0080] S14. Construct a Mel filter based on the inverse conversion result of the conversion relationship between the Mel frequency and the preceding frequency. Use the Mel filter to filter the frequency domain information obtained in S13. The expression is:
[0081]
[0082] Where i is the Mel range parameter, f[i-1], f[i], and f[i+1] are all linear frequency response functions, and H[i,k] is the filter value;
[0083]
[0084] Where M[m,i] is the Mel filter spectrum;
[0085] S15. The cosine transform method is used to reduce the dimensionality of the Mel filter spectrum. The expression is:
[0086] E[m,i]=log(M[m,i]);
[0087] Where E[m,i] is the dimensionality reduction result;
[0088] S16. The DCT algorithm is used to eliminate the correlation between Mel spectra and extract spectral features to obtain the music feature coefficients, the expression of which is:
[0089]
[0090] Among them, MFCC music[m,o] represents the musical characteristic coefficients, and o is a set of cepstral coefficients.
[0091] In some embodiments of this application, such as Figure 3 As shown in Figure S2, the Mel frequency cepstral coefficients extracted from the EEG signal include:
[0092] S21. Extract the Mel-frequency cepstral coefficients of the EEG signal, the expression of which is:
[0093]
[0094] Among them, MFCC EEG [m,o] represents the extracted Mel frequency cepstral coefficients, P EEG [m,k] represents the frequency domain information of the EEG signal.
[0095] In some embodiments of this application, in S2, optimizing and updating the order of the Mel frequency cepstral coefficients of the EEG signal so that the combined error value between the reconstructed EEG feature signal of the Mel frequency cepstral coefficients and the original EEG signal meets preset conditions includes:
[0096] S22. Perform inverse DCT, exponential operation, inverse Mel filtering, inverse FFT and overlap summation on the Mel frequency cepstral coefficients of the EEG signal in sequence to obtain the EEG feature reconstruction signal;
[0097] S23. The combined error value is obtained based on the time-domain error and the frequency-domain error, expressed as follows:
[0098]
[0099]
[0100] in, For time-domain error, For frequency domain error, The total error value is given by T, where T is the time domain range and K is the frequency domain range. EEG The raw EEG signal, X reconstructed PSD(X) is a signal reconstructed from EEG features. EEG ) represents the power spectral density of the original EEG signal, PSD(X) reconstructed ) represents the power spectral density of the signal reconstructed from EEG features, w time For time-domain error weights, w freq For frequency domain error weights;
[0101] S24. Iteratively update the order of the reconstructed signal based on the comprehensive error value; if the reconstruction error of the current order is lower than the preset error, update the order and reconstruction error; if the reconstruction error of the current order is higher than the preset error, reduce the search step size and iterate again.
[0102] In some embodiments of this application, in S3, the FastDTW algorithm is used to align the music feature coefficients and the reconstructed EEG feature signals in the feature dimension and time axis, and the alignment results include:
[0103] Because the MFCC order of music signals and EEG signals is inconsistent (music signals are usually fixed at order 13, while the order of EEG signals needs to be dynamically determined through adaptive search), this leads to problems of dimensional differences and dynamic temporal variations when aligning the two signals' features. To address this challenge, this invention introduces the FastDTW (Fast Dynamic Time Warping) method, which can achieve efficient alignment in both the feature dimension and the time axis. FastDTW significantly reduces the computational complexity of traditional DTW by initially aligning features on the downsampled low-resolution path and then gradually refining them to the high-resolution path, while maintaining high alignment accuracy. Figure 4 As shown.
[0104] S31. Obtain the similarity value between the audio signal and the EEG signal that satisfy the preset constraints, wherein the preset constraints include at least boundary constraints, continuity constraints, and monotonicity constraints.
[0105] The expression for similarity value is:
[0106]
[0107] Where dist is the similarity value, w c Here, C represents the number of constraints.
[0108] It's important to note that DTW requires calculating the entire cumulative distance matrix, resulting in a space and time complexity of O(IJ), which is unsuitable for real-time systems. fastDTW is an approximate algorithm designed for fast DTW distance calculation. It uses a multi-resolution strategy combined with local search to reduce computational complexity, making it more suitable for handling long sequences. First, a multi-scale representation is generated through downsampling, halving the sequence length each time. This allows for a quick finding of the approximate path at a coarse-grained level. Specifically, the original sequence is decomposed into multiple resolutions; then, standard DTW is performed at the coarsest resolution to calculate the initial path, which is computationally inexpensive due to the short sequence. Next, the low-resolution path is mapped to a high-resolution space using a projection function P, providing a good initial estimate for subsequent fine-tuning. Given low-resolution path points, the path in the high-resolution space is calculated; this process completes a recursive process: first recursively recursively to the coarsest resolution, then progressively optimizing upwards to the original resolution.
[0109] Finally, the optimal path is obtained through local adjustments using radius constraints. The search is performed within a strip region surrounding the projected path, avoiding a global search.
[0110] S32. For the music feature coefficients and EEG feature reconstructed signals, a downsampling algorithm is used to generate a multi-scale representation. The sequence length is halved each time it is generated, and multi-resolution decomposition is performed. Standard DTW is performed on the low resolution to calculate the initial low-resolution path.
[0111] S33. The initial low-resolution path is mapped to the high-resolution space through the projection function, and the original resolution path is obtained by optimizing it step by step based on the resolution level.
[0112] S34, such as Figures 5-6 As shown, a radius constraint algorithm is used for local adjustment to obtain the optimal path at the original resolution. The FastDTW algorithm is then used for cumulative calculation to obtain the distance matrix, and the alignment path is calculated based on this distance matrix. The expression is as follows:
[0113] path'={(i1,j1),…(i Z ,j Z )};
[0114] Where path' is the alignment path, (i1,j1) is the element value in the first row and first column of the distance matrix, (i Z ,j Z ) represents the element value in the Z-th row and Z-th column of the distance matrix.
[0115] In some embodiments of this application, in step S4, identifying the degree of music enjoyment of the person to be identified based on the alignment result includes:
[0116] The current music signal and the EEG signal of the person to be identified are acquired. Based on steps S1-S3, the current music signal and the EEG signal of the person to be identified are processed to obtain the current signal alignment path. The degree of music enjoyment of the person to be identified is identified based on the current signal alignment path.
[0117] The advantages and beneficial effects of this invention compared to the prior art are:
[0118] 1. This invention proposes an adaptive search strategy to optimize the order of MFCCs in EEG signals. This strategy can dynamically optimize the extraction dimension of MFCC order based on the specific characteristics of the signal. In particular, compared with existing fixed-order methods, the adaptive dynamic search strategy balances the accuracy and redundancy of MFCC order, thereby improving the effectiveness and robustness of signal feature extraction.
[0119] 2. This invention proposes a novel music-to-brainwave (MEFF) fusion mechanism based on FastDTW to analyze the synchronicity of music-to-brainwave signals for use in tasks involving the identification of participants' level of musical enjoyment. This mechanism uses the FastDTW algorithm to solve the problem of efficient alignment of the MFCCs feature dimensions of the two signals. Compared to results using only EEG features, fusing music and EEG features significantly improves recognition accuracy.
[0120] 3. This invention improves the interpretability of results by analyzing the recognition performance of brain regions and the correlation between the proposed features and the music enjoyment rating. This work provides a practical application reference for improving mental health interventions through personalized music therapy, and also offers suggestions for lead optimization, potentially promoting the widespread adoption of wearable EEG sound intervention systems.
[0121] The embodiments of the present invention will be described in detail below with reference to specific examples.
[0122] This invention utilizes electroencephalogram (EEG) data collected from depressed and healthy participants listening to music using a 64-lead acquisition device from BrainProducts, Germany. The selection of music materials was based on recommendations from experienced music therapists. For the EEG data, a Butterworth high-pass filter (order=8) and a Chebyshev type I low-pass filter (order=16) were first applied to filter the data from 0.3 to 50 Hz. The filtered data was then downsampled to 125 Hz. This invention removed the EOG components used to track vertical and horizontal eye movements, retaining 62 electrodes for further analysis. Next, independent component analysis was used to remove eye movement and cardiac artifacts from the data. All data processing was performed using Matlab 2020b and the EEGLAB toolbox. Each participant, after preprocessing, contained a 62*T (electrode*time) matrix, where T varied according to the length of the music. To maintain consistency, the data was aggregated for each song. Therefore, the data format for each song was a three-dimensional matrix of size 62*T*15 (electrode*time*participant). Similarly, this invention follows... Figure 7 The electrodes were divided into 5 brain regions.
[0123] Music-EEG Feature Extraction and Fusion: Based on the preceding introduction, this invention first extracts the Mel-spectral coefficients of the music signal, empirically selecting N... coeff =13, meaning the first 13 order coefficients are used for subsequent calculations. Thus, this invention obtains the music feature as music * number of frames * order (10 * m * 13). Furthermore, this invention uses an adaptive search method to extract the Mel-Cepstral Coefficients (MCCs) of the EEG signal, resulting in an EEG MFCC feature dimension of brain region * subject * song * order (0 * m * 13). bestThe matrix represents the number of frames in a 5D matrix. Because the MFCC order of music signals and EEG signals is inconsistent (music signals are typically fixed at order 13, while the order of EEG signals needs to be dynamically determined through adaptive search), feature alignment between the two signals faces challenges due to dimensional differences and dynamic temporal variations. To address this challenge, this invention introduces the FastDTW (Fast Dynamic Time Warping) method, which achieves efficient alignment along the feature dimension and time axis. FastDTW significantly reduces the computational complexity of traditional DTW while maintaining high alignment accuracy by initially aligning features on the downsampled low-resolution path and gradually refining them to the high-resolution path.
[0124] Participant Music Enjoyment Level Recognition: After extracting adaptive MFCC features from EEG signals and addressing the alignment issue with music signal features, this invention further validates the effectiveness of the proposed features in recognizing participants' music enjoyment levels. Specifically, this invention employs eight classifiers to classify the extracted features, covering five classic machine learning methods and three deep neural network models. The machine learning methods include Support Vector Machine (SVM), k-Nearest Neighbors (k-NN), Extreme Gradient Boosting (XGBoost), Random Forest (RF), and Lightweight Gradient Boosting (LightGBM), which achieve efficient feature classification with low computational overhead. Simultaneously, this invention also introduces deep learning models such as Convolutional Neural Networks (CNN), VGGNet, and ResNet to fully explore the nonlinear relationships and high-order semantic information of the features, further improving classification performance. To objectively evaluate the proposed features, several metrics were calculated to assess the classification results: performance metrics included classification accuracy (Acc) and area under the receiver operating characteristic curve (AUC). All experiments were implemented on an Intel(R) Core(TM) i5-10400F CPU and an NVIDIA GeForce RTX 4090 @ 24GB VRAM GPU in a Python 3.9 environment. The deep learning model used PyTorch with CUDA 12.4 support.
[0125] This invention employs an intra-subject cross-validation method, dividing the data for each individual subject. Data from the same subject is randomly divided into training and test sets to evaluate feature performance within the individual. A stratified KFold 5-fold cross-validation is used, maintaining a proportion of samples from each class in each fold that is roughly consistent with the proportion in the entire dataset. The average of these five accuracies is then calculated as the final accuracy for that subject. This validation method eliminates the influence of inter-individual differences, focusing on the consistency of features within a single subject and their ability to distinguish levels of musical enjoyment. The results of this invention demonstrate the consistency of the proposed features within a single subject and their ability to distinguish levels of musical enjoyment (73.49% accuracy for healthy participants and 78.73% accuracy for depressed patients). Overall, the frontal lobe showed significantly higher classification performance than other brain regions on both datasets. Deep learning-based models performed particularly well, exhibiting strong feature extraction and classification capabilities. Specifically, there were significant differences in classification performance across different brain regions. In particular, the frontal and temporal lobes generally performed better than the central, parietal, and occipital lobes. Furthermore, the overall classification performance of depressed patients was higher than that of healthy participants, especially with deep learning models. For example, in the healthy group, the VGGNet model for the temporal lobe region achieved the best classification performance (Acc: 78.73 ± 1).
[0126] The highest accuracy rate (Acc: 5.97%; AUC: 83.69±8.28) was observed in the frontal lobe, while the lowest k-NN model classification performance was observed in the occipital lobe (Acc: 64.52±5.08%; AUC: 70.01±6.64). In the depression group, the ResNet model performed best in the frontal lobe (Acc: 78.73±5.97%; AUC: 83.69±8.28), while the lowest k-NN model classification performance was observed in the central lobe (Acc: 70.61±6.23%; AUC: 78.16±8.22).
[0127] To compare the advantages of adaptive dynamic search methods, this invention uses the traditional coefficient cross-validation method on EEG signals to determine the coefficient order of the MFCC that yields the best recognition performance. The distribution results of the MFCC order of EEG signals are as follows: Figure 8 As shown. Figure 8 (a) shows the results of traditional coefficient cross-validation, which shows that the sum of classification accuracy for all brain regions is highest when the order of EEG MFCC is 8. Figure 8(b) Frequency statistics of the order in the adaptive coefficient search of MFCC. The order distribution does not exhibit a specific value but shows a significant main peak structure, concentrated in the 7th-10th order MFCC coefficient range. The overall distribution shows a typical right-skewed feature, with the frequency rapidly decaying to near 0 in the high-order coefficient region (>12th order), indicating that the contribution of high-order MFCC coefficients to the expression of EEG features is relatively small. Furthermore, to verify the effectiveness of the fused features, this invention compared and analyzed the MFCC of EEG features with features fused with music signals. Then, a 5-fold cross-validation was used to divide the dataset, and the VGGNet classifier was used to calculate the classification performance index as the average accuracy. The results show that the adaptive dynamic coefficient search method outperforms the traditional coefficient cross-validation method in all brain regions in both datasets, especially in the parietal lobe region, where the accuracy improvement is significant (healthy group: 21.6%, depressed group: 21.2%). The increasing trend is more pronounced in the depressed group. Furthermore, in Figure (b), feature fusion significantly improved classification performance compared to experiments using only EEG signals, particularly in the frontal and parietal regions, with an overall accuracy improvement of approximately 20%, including the frontal lobe (healthy group: 24.7%, datdaset2: 19.2%) and the parietal lobe (healthy group: 23.1%, datdaset2: 20.7%).
[0128] Correlation analysis of fusion features and participants' music enjoyment ratings: To further verify the correlation between the extracted features and participants' music enjoyment levels, this invention calculated the Spearman correlation coefficient between features and subjective ratings. Spearman correlation is a non-parametric statistical method that effectively measures the monotonic correlation between two variables and is suitable for scenarios where feature distributions are non-linear and ratings are non-continuous variables. In the experiment, this invention calculated the Spearman correlation between the optimal path features of five brain regions and participants' ratings of their respective music enjoyment levels to assess the strength of the correlation between features and music enjoyment levels. The results showed that the correlations in all brain regions reached a significant level (p = 0.00), indicating a statistically significant association between these brain region features and ratings. In the healthy control group, different brain region characteristics showed a positive correlation with scores: the frontal lobe region showed the highest correlation (r = 0.39), indicating a strong positive correlation between features in this region and scores; the central and temporal lobes showed lower correlations (r = 0.34 and r = 0.31, respectively), but still showed a certain positive correlation trend; the parietal and occipital lobes showed slightly higher correlations than the temporal lobe region (r = 0.35 and r = 0.37, respectively). In the depression group, all brain region characteristics showed a negative correlation with scores: the temporal and parietal lobes showed the strongest negative correlations (r = -0.47 and r = -0.46, respectively), indicating that features in these regions decreased as scores increased. The frontal and central lobes showed the second strongest correlations (r = -0.45), showing a significant negative correlation. The occipital lobe region showed the lowest negative correlation (r = -0.37), consistent with the healthy control group.
[0129] In summary, the results of this invention demonstrate that the frontal and temporal lobes exhibited the best recognition performance in both subject-relevant and subject-irrelevant scenarios across two datasets. The temporal lobe, particularly the auditory cortex, showed higher activity when people listened to pleasant music, reflecting enhanced auditory processing and appreciation of musical elements. Research indicates that the right temporal lobe is associated with emotional responses to music, with higher levels of enjoyment correlated with greater activation. The frontal lobe, especially the nucleus accumbens, plays a crucial role in the brain's reward system; increased enjoyment leads to enhanced activity associated with well-being and reward processing. Significant synchronicity between frontal and temporal lobe regions during music-induced pleasure reflects their synergistic function in processing auditory information, emotional responses, and familiarity effects, contributing to a richer musical experience. Furthermore, EEG-music synchronicity characteristics were positively correlated with music enjoyment scores in healthy individuals, but negatively correlated in depressed individuals. This highlights the impact of emotional disorders on the neural processing of music experience. Research suggests that depressed individuals may not exhibit the same levels of neural synchronicity or emotional response as normal listeners, resulting in a decreased ability to derive pleasure from music. Higher EEG synchronicity can predict an individual's musical preferences, indicating that the level of neural response synchronization increases with increased enjoyment. Exploring the use of neuroimaging techniques to monitor brain activity during music listening can provide insights into the specific mechanisms by which different types of music improve depressive symptoms. This can contribute to the development of precise and personalized music therapy methods, improving treatment outcomes for patients with depression. Utilizing the similarities and differences in neural characteristics of music appreciation for the auxiliary diagnosis and personalized music therapy of depression has significant clinical and research value, providing new ideas and methods for improving the quality of life of patients with depression.
[0130] In this application, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. In case of any inconsistency, the meaning set forth in this specification or derived from the content described herein shall prevail. Furthermore, the terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit the scope of this application.
[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.
Claims
1. A method for recognizing the degree of music enjoyment based on music-EEG feature fusion, characterized in that, Includes the following steps: S1. Extract the Mel frequency cepstral coefficients of the music signal, preprocess the music signal and perform fast Fourier transform to obtain the spectral feature values of the music signal, and then perform filtering, dimensionality reduction and decorrelation processing on the spectral feature values of the music signal in sequence to obtain the music feature coefficients. S2. Extract the Mel frequency cepstral coefficients of the EEG signal, optimize and update the order of the Mel frequency cepstral coefficients of the EEG signal, so that the comprehensive error value between the EEG feature reconstruction signal of the EEG signal Mel frequency cepstral coefficients and the original EEG signal meets the preset conditions. S3. The FastDTW algorithm is used to align the music feature coefficients and the reconstructed EEG feature signals in the feature dimension and time axis to obtain the alignment result; S4. Identify the degree of music enjoyment of the person being identified based on the alignment results.
2. The method for recognizing the degree of music enjoyment based on music-EEG feature fusion according to claim 1, characterized in that, In step S1, the preprocessing and fast Fourier transform of the Mel frequency cepstral coefficients of the music signal include: S11. Perform pre-emphasis processing on the music signal to obtain the Mel frequency cepstral coefficients after the high frequency components are enhanced; S12. The pre-emphasized signal is sequentially framed and windowed to reduce spectrum leakage; S13. Perform a Fast Fourier Transform on the windowed value to obtain the frequency domain information of the music signal, expressed as: Where s[m,n] is the window value, S[m,k] is the Fast Fourier Transform value of the window value, n is the window length limit, N is the window length, m is the frame index, k is the transform parameter, and N FFT For the window length of the Fast Fourier Transform, P music [m,k] represents the frequency domain information of the music signal.
3. The method for recognizing the degree of music enjoyment based on music-EEG feature fusion according to claim 2, characterized in that, In step S1, the spectral feature values of the music signal are sequentially filtered, reduced in dimensionality, and decorrelated to obtain music feature coefficients, including: S14. Construct a Mel filter based on the inverse conversion result of the conversion relationship between the Mel frequency and the preceding frequency. Use the Mel filter to filter the frequency domain information obtained in S13. The expression is: Where i is the Mel range parameter, f[i-1], f[i], and f[i+1] are all linear frequency response functions, and H[i,k] is the filter value; Where M[m,i] is the Mel filter spectrum; S15. The cosine transform method is used to reduce the dimensionality of the Mel filter spectrum. The expression is: E[m,i]=log(M[m,i]); Where E[m,i] is the dimensionality reduction result; S16. The DCT algorithm is used to eliminate the correlation between Mel spectra and extract spectral features to obtain the music feature coefficients, the expression of which is: Among them, MFCC music [m,o] represents the musical characteristic coefficients, and o is a set of cepstral coefficients.
4. The method for identifying the level of music enjoyment based on music-EEG feature fusion according to claim 3, characterized in that, In step S2, the Mel frequency cepstral coefficients of the extracted EEG signal include: S21. Extract the Mel-frequency cepstral coefficients of the EEG signal, the expression of which is: Among them, MFCC EEG [m,o] represents the extracted Mel frequency cepstral coefficients, P EEG [m,k] represents the frequency domain information of the EEG signal.
5. The method for identifying the level of music enjoyment based on music-EEG feature fusion according to claim 4, characterized in that, In step S2, the order of the Mel-frequency cepstral coefficients of the EEG signal is optimized and updated so that the combined error value between the reconstructed EEG feature signal of the Mel-frequency cepstral coefficients and the original EEG signal meets the preset conditions, including: S22. Perform inverse DCT, exponential operation, inverse Mel filtering, inverse FFT and overlap summation on the Mel frequency cepstral coefficients of the EEG signal in sequence to obtain the EEG feature reconstruction signal; S23. The combined error value is obtained based on the time-domain error and the frequency-domain error, expressed as follows: in, For time-domain error, For frequency domain error, The total error value is given by T, where T is the time domain range and K is the frequency domain range. EEG The raw EEG signal, X reconstructed PSD(X) is a signal reconstructed from EEG features. EEG ) represents the power spectral density of the original EEG signal, PSD(X) reconstructed ) represents the power spectral density of the signal reconstructed from EEG features, w time For time-domain error weights, w freq For frequency domain error weights; S24. Iteratively update the order of the reconstructed signal based on the comprehensive error value; if the reconstruction error of the current order is lower than the preset error, update the order and reconstruction error; if the reconstruction error of the current order is higher than the preset error, reduce the search step size and iterate again.
6. The method for identifying the level of music enjoyment based on music-EEG feature fusion according to claim 5, characterized in that, In step S3, the FastDTW algorithm is used to align the music feature coefficients and the reconstructed EEG feature signals in the feature dimension and time axis, and the alignment results include: S31. Obtain the similarity value between the audio signal and the EEG signal that satisfy the preset constraints, wherein the preset constraints include at least boundary constraints, continuity constraints, and monotonicity constraints. The expression for similarity value is: Where dist is the similarity value, w c Here, C represents the number of constraints. S32. For the music feature coefficients and EEG feature reconstructed signals, a downsampling algorithm is used to generate a multi-scale representation. The sequence length is halved each time it is generated, and multi-resolution decomposition is performed. Standard DTW is performed on the low resolution to calculate the initial low-resolution path. S33. The initial low-resolution path is mapped to the high-resolution space through the projection function, and the original resolution path is obtained by optimizing it step by step based on the resolution level. S34. A radius constraint algorithm is used for local adjustment to obtain the optimal path at the original resolution. The FastDTW algorithm is then used for cumulative calculation to obtain the distance matrix. Based on this distance matrix, the alignment path is calculated, expressed as: path'={(i1,j1),…(i Z ,j Z )}; Where path' is the alignment path, (i1,j1) is the element value in the first row and first column of the distance matrix, (i Z ,j Z ) represents the element value in the Z-th row and Z-th column of the distance matrix.
7. The method for recognizing the degree of music enjoyment based on music-EEG feature fusion according to claim 6, characterized in that, In step S4, identifying the degree of music enjoyment of the person to be identified based on the alignment result includes: The current music signal and the EEG signal of the person to be identified are acquired. Based on steps S1-S3, the current music signal and the EEG signal of the person to be identified are processed to obtain the current signal alignment path. The degree of music enjoyment of the person to be identified is identified based on the current signal alignment path.