An adaptive music intervention system based on multi-modal physiological feedback

Abstract

The application discloses a kind of adaptive music intervention systems based on multi-modal physiological feedback, belong to the monitoring and intervention technical field of emotional state, including multi-modal emotional state evaluation module and adaptive music intervention module, multi-modal emotional state evaluation module is by collecting the multi-modal physiological information of user, and multi-modal data is extracted and fused, according to the fusion data, emotional state is evaluated;Adaptive music intervention module generates the music feature matched with current emotion according to the emotional state of evaluation, music feature and the preference data of user, and according to music feature, music is selected to intervene user.The application uses the above method, wearable prefrontal three-lead electroencephalogram equipment and smart bracelet are combined, physiological signals of central nervous system and autonomic nervous system are collected and analyzed in real time, automatically adjust music intervention, provide side effect-free, personalized emotional regulation scheme, form closed-loop mental state control mechanism.

Description

Technical Field

[0001] This invention relates to the field of emotional state monitoring and intervention technology, and in particular to an adaptive music intervention system based on multimodal physiological feedback. Background Technology

[0002] With increasing global economic pressure and a faster pace of life, mental health issues such as depression are becoming increasingly serious. Globally, there are approximately 280 million people suffering from depressive disorders, making it a major public health problem affecting mental health worldwide. Of particular concern is the trend of depressive disorders occurring at younger ages, with more and more teenagers experiencing these issues, further exacerbating the severity of this social problem.

[0003] Patients with depressive disorders typically exhibit increased negative emotions and decreased positive emotions, which is closely related to their impaired emotion regulation abilities. Studies have shown that depressed individuals struggle to employ effective emotion regulation strategies when facing emotional difficulties. Neuroscience research has found that the responses of brain regions such as the amygdala and prefrontal cortex in depressed individuals differ from those in healthy individuals when processing emotional stimuli. These brain regions play a crucial role in emotion regulation, and abnormalities in their function may be directly related to difficulties in emotion regulation.

[0004] Emotion regulation involves complex physiological and psychological mechanisms. Research in cognitive science and psychophysiology shows that the perception and regulation of emotion involves the combined action of the central nervous system (CNS) and the peripheral nervous system (PNS). When the external environment changes or stimuli appear, the sensory organs of the PNS rapidly transmit information to different levels of the brain, activating the mechanisms of emotion perception and regulation. Brain structures closely related to emotional responses, such as the hypothalamus, amygdala, and prefrontal cortex, integrate this information to form emotional experiences, which are then triggered by efferent neural pathways to express emotions in the body. This emotion regulation process not only activates the central nervous system but also triggers functional changes in the PNS, especially the autonomic nervous system (ANS). The ANS includes the sympathetic nervous system (SNS) and the parasympathetic nervous system (PNS), whose activity can affect various physiological functions such as heartbeat, respiration, blood pressure, digestion, and metabolism. Therefore, acquiring central nervous system physiological signals (such as electroencephalogram) and autonomic nervous system physiological signals (such as electrocardiogram and skin conductance) during the emotion regulation process provides an objective basis for analyzing emotional states.

[0005] Numerous studies have demonstrated the effectiveness of music therapy in managing mood disorders, particularly depressive disorders. By modulating the autonomic nervous system, music can effectively improve an individual's emotional state and enhance their mental health.

[0006] Therefore, there is an urgent need to develop an adaptive music intervention system based on multimodal physiological feedback to monitor and intervene in the emotional state of patients with depressive disorders. Summary of the Invention

[0007] The purpose of this invention is to provide an adaptive music intervention system based on multimodal physiological feedback, which combines a wearable three-lead prefrontal cortex EEG device with a smart bracelet to collect and analyze physiological signals of the central nervous system and autonomic nervous system in real time, automatically adjust music intervention, provide a personalized emotion regulation program without side effects, and form a closed-loop mental state regulation mechanism.

[0008] To achieve the above objectives, this invention provides an adaptive music intervention system based on multimodal physiological feedback, comprising a multimodal emotional state assessment module and an adaptive music intervention module. The multimodal emotional state assessment module collects the user's multimodal physiological information, performs multi-scale feature extraction and multimodal data fusion on the multimodal data, and assesses the emotional state based on the fused data. The adaptive music intervention module generates music features that match the current emotion based on the assessed emotional state, music features, and the user's preference data, and selects music to intervene with the user based on the music features.

[0009] Preferably, the method for implementing the multimodal emotion state assessment module includes:

[0010] The wearable three-lead prefrontal cortex EEG device and smart bracelet can collect users' multimodal physiological signals in real time, including EEG, heart rate, skin conductance and blood oxygen saturation signals.

[0011] The raw signals of the acquired multimodal data are preprocessed;

[0012] A unified time window for multimodal data is adopted, and a hybrid domain feature extraction strategy is used to extract features from the preprocessed multimodal data signals. Then, all modal features after feature extraction are normalized.

[0013] Linear transformation and contrastive learning strategies are used to align the feature-extracted multimodal data in terms of dimension and semantics, and weighted fusion is performed through an attention mechanism;

[0014] An emotion classification model consisting of two fully connected layers identifies the fused multimodal features and classifies emotional states into positive or negative categories.

[0015] Preferably, the feature extraction strategy using a hybrid domain to extract features from the preprocessed multimodal data signal includes:

[0016] Temporal feature extraction: First, extract the mean, variance, and extreme values ​​of EEG, heart rate, skin conductance, and blood oxygen saturation signals, and then extract the heart rate variability features of the heart rate signal;

[0017] Frequency domain feature extraction: Fourier transform is used to convert the EEG signal from the time domain to frequency domain features, which include frequency domain energy, dominant frequency, and frequency domain energy ratio;

[0018] Time-frequency domain feature extraction: Combining time and frequency information, short-time Fourier transform and wavelet transform are used to perform time-frequency analysis on EEG signals. Short-time Fourier transform generates a time-frequency representation and extracts the frequency distribution of EEG signals at different times. Wavelet transform performs multi-scale analysis on EEG signals by scaling and shifting the mother wavelet function to extract the local frequency information of the signal and capture the instantaneous spectral characteristics during emotional changes. Based on the instantaneous spectral characteristics, the instantaneous frequency of the signal at a certain moment is extracted.

[0019] Preferably, the step of aligning the feature-extracted multimodal data in terms of dimensionality and semantics using linear transformation and contrastive learning strategies includes:

[0020] A fully connected layer is used to perform a linear transformation on the modal features, mapping each modal feature to the same vector feature space. The feature vector after the fully connected layer is represented as:

[0021] P i =W i F i +b i ;

[0022] Where i∈{EEG,HR,EDA,SpO2} represents the mode, W i Let b represent the weight matrix of the fully connected layer. i F represents the bias term. i Indicates modal feature input;

[0023] A contrastive learning framework is used for semantic alignment of features from different modalities. First, the cosine similarity of each pair of modal (i,j) features under the same emotional state is calculated, using the following formula:

[0024]

[0025] Among them, P i k and Let i and j represent the feature projections of mode i and mode j, respectively, and k represent the same time sampling point;

[0026] Secondly, we increase the feature similarity of positive sample pairs to ensure semantic alignment and decrease the feature similarity of negative sample pairs to ensure the discriminative power of emotional states in the feature space. Positive sample pairs are modal feature pairs under the same emotional state, while negative sample pairs are modal feature pairs under different emotional states.

[0027] Then, the contrastive learning loss between each pair of modes is calculated. The loss function for mode i and mode j is expressed as:

[0028]

[0029] Wherein, the numerator represents the similarity of positive sample pairs under the same emotional state, the denominator represents the similarity of negative sample pairs under different emotional states, and τ represents the temperature parameter, which is used to adjust the contrast of characteristic similarity and enhance the model's sensitivity to positive and negative samples.

[0030] Finally, the final contrastive learning loss function is calculated. This final contrastive learning loss function is the sum of the losses between all mode pairs, expressed as:

[0031]

[0032] Where N is the total number of modes.

[0033] Preferably, the weighted fusion of features from different modalities using an attention mechanism includes:

[0034] First, calculate the attention weight for each modality feature, expressed as:

[0035]

[0036] Among them, W α α is the learning parameter for attention weights. i P represents the attention weights for mode i. i P represents the projection of mode i. j Represents the projection of mode j;

[0037] Then, feature fusion is performed, and the fused feature vector F F Represented as:

[0038]

[0039] Preferably, the adaptive music intervention module implementation method includes:

[0040] Construct a CVAE model, which includes a decoder and an encoder;

[0041] The data on emotional state, music features, and user preferences are preprocessed, and the processed data is then input into the CVAE model.

[0042] The encoder receives various preprocessed data, maps the input data to the latent space through a neural network, generates the mean and variance of the latent variables, and outputs the distribution of the latent variables based on the latent variables.

[0043] The decoder takes latent variables and inputs the user's emotional and behavioral conditions to generate a new musical feature;

[0044] The generated music features are matched with actual music features in the music library through similarity retrieval. The music is sorted according to similarity and a personalized recommendation list that matches the user's emotional state is generated. The personalized recommendation list is dynamically updated according to the real-time changes in the user's emotional state and behavior.

[0045] Preferably, the data preprocessing includes:

[0046] Extract the current emotional state assessed by the current multimodal emotional state assessment module and represent it as an emotional distribution vector E. user Each emotion category corresponds to a probability value; long-term music preference information is extracted from users' historical behavior data, including users' listening records, ratings, and skip rates, and represented as a user personalized preference vector P. user Extract feature information from each piece of music in the music library, including pitch, rhythm, melody, and emotional tags, and represent it as music feature x.

[0047] Preferably, the step of mapping input data to a latent space using a neural network, generating the mean and variance of latent variables, and outputting the distribution of latent variables based on the latent variables includes:

[0048] The encoder maps input data to a latent space using a neural network, generating the mean μ of the latent variable z. φ (x,c) and variance Σ φ (x,c), where c=[E user ,P user [] indicates conditional input;

[0049] The encoder outputs the distribution of the latent variables based on the latent variables, which is represented as:

[0050]

[0051] Using the reparameterization technique, the latent variables are represented as:

[0052] z = μ φ (x,c)+∈·Σ φ (x,c);

[0053] Where ∈ represents the standard normal distribution The noise in the sampled data.

[0054] Preferably, the decoder takes latent variables and inputs the user's emotional and behavioral conditions to generate a new musical feature, including:

[0055] The decoder generates musical features that match the user's emotions and preferences by modeling latent variables and conditional inputs, represented as:

[0056]

[0057] Where, p θ (x′|z,c) represents the distribution of music features generated by the decoder, μ θ (z,c) and Σ θ (z,c) represent the mean and variance of the generated features, respectively;

[0058] The reconstruction loss and KL divergence loss functions are used to calculate the loss to ensure that the generated music feature x' meets the needs of the user's current emotional state and historical preferences;

[0059] The reconstruction loss metric is expressed as:

[0060]

[0061] The KL divergence loss function is expressed as:

[0062]

[0063] Where p(z) represents the standard normal distribution, D KL This represents the KL divergence between music characteristics and the user's current emotional state and historical preference needs.

[0064] The total loss function is expressed as:

[0065]

[0066] Where λ represents the hyperparameter that adjusts the reconstruction loss and the KL divergence weights.

[0067] Therefore, the adaptive music intervention system based on multimodal physiological feedback described above, as used in this invention, has the following beneficial effects:

[0068] (1) By combining a wearable three-lead prefrontal EEG acquisition device and a smart bracelet, the activity of the prefrontal cortex emotional response area of ​​the brain can be monitored in real time to obtain signals such as heart rate and blood oxygen, and to comprehensively assess the user's emotional state.

[0069] (2) By fusing multimodal physiological data such as EEG, heart rate, skin conductance and blood oxygen saturation, and combining the modal feature alignment method of contrastive learning, high-precision emotional state recognition was achieved.

[0070] (3) By using attention mechanism weighted fusion, the contribution of each modality to emotion recognition is dynamically adjusted, ensuring the robustness and flexibility of the system under complex emotional states.

[0071] (4) The adaptive music intervention module based on CVAE can generate personalized music recommendations based on the user's current emotional state and historical behavior data, thereby effectively helping the user to regulate their emotions and improving the user experience and the system's intelligence.

[0072] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0073] Figure 1 This is a system architecture diagram of an embodiment of the present invention;

[0074] Figure 2 This is a detailed flowchart of the multimodal emotion state assessment module according to an embodiment of the present invention;

[0075] Figure 3 This is a detailed flowchart of the adaptive music intervention module according to an embodiment of the present invention;

[0076] Figure 4 This is a flowchart illustrating the system implementation of an embodiment of the present invention. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0078] Example

[0079] Reference Figure 1-3 The present invention provides an adaptive music intervention system based on multimodal physiological feedback, including a multimodal emotional state assessment module and an adaptive music intervention module.

[0080] The multimodal emotion state assessment module collects users' multimodal physiological information and performs multi-scale feature extraction and multimodal data fusion on the multimodal data to assess the emotional state. The adaptive music intervention module generates music features that match the current emotion based on the assessed emotional state, music features, and user preference data, and selects music to intervene with the user based on these music features.

[0081] The implementation methods for the multimodal emotion state assessment module include:

[0082] Physiological signal acquisition.

[0083] A wearable three-lead prefrontal cortex EEG device and a smart bracelet are used to collect multimodal physiological signals from users in real time, including electroencephalogram (EEG), heart rate (HR), electrodermal activity (EDA), and blood oxygen saturation (SpO2) signals. The wearable three-lead prefrontal cortex EEG device includes three channels (Fp1, Fp2, and Fp2). EEG signals are collected at a sampling frequency of 250Hz using a three-channel non-invasive sensor, while HR, EDA, and SpO2 signals are collected at a frequency of 1Hz using an integrated sensor built into the smart bracelet. These signals can reflect the user's physiological changes and thus reveal emotional fluctuations. The collected physiological signals are set as follows:

[0084]

[0085] Where, x EEG ,x HR ,x EDA , These represent physiological signals of different modalities.

[0086] The raw signals of the acquired multimodal data are preprocessed.

[0087] The acquired raw signals may contain noise and artifacts, which are then preprocessed. The EEG signal is bandpass filtered (0.5Hz to 45Hz) to remove low-frequency interference (such as eye-tracking artifacts) and high-frequency noise (such as power supply noise). The HR signal is smoothed to remove transient fluctuations in the heartbeat interval.

[0088] Feature extraction.

[0089] A unified time window is used to ensure that all modal signals are synchronized within the same time window, thus achieving temporal alignment of the multimodal data signals. A hybrid-domain feature extraction strategy is employed to extract features from the preprocessed multimodal data signals, and all modal features after extraction are normalized. Specifically, this includes:

[0090] Physiological signals of different modalities are analyzed according to a set time window T. w Alignment is performed to ensure all modal signals are synchronized within the same time window. For high-frequency signals (EEG), each T... w Internal features are extracted, such as mean, variance, and frequency domain energy. For low-frequency signals (such as HR, EDA, and SpO2), T is used directly. w Statistical processing is performed on data from multiple seconds.

[0091] Extracting multimodal data features includes extracting time-domain, frequency-domain, and time-frequency-domain features of user EEG, HR, EDA, and SpO2 signals.

[0092] Temporal characteristics:

[0093] In the time domain, statistical features reflecting the fluctuation characteristics and overall trend of the signal are directly extracted from the time series of the signal. Time-domain features help capture the overall changes in physiological signals during emotional fluctuations. Specifically, this involves extracting the mean, variance, and extreme values ​​of EEG, heart rate, skin conductance, and blood oxygen saturation signals, and then further extracting the heart rate variability (HRV) feature of the heart rate signal.

[0094] Frequency domain characteristics:

[0095] Fourier transform is used to convert EEG signals from the time domain to the frequency domain, extracting energy features at different frequency bands. Frequency domain features include:

[0096] Frequency domain energy E f The energy in specific frequency bands (α, β, and θ bands are associated with emotions such as relaxation, tension, and alertness, respectively) is expressed as... Where X(f) represents the Fourier transform result of the signal, and f1 and f2 represent the start and end frequencies of the frequency band.

[0097] main frequency f peak , is the frequency component with the highest signal energy, representing the most important frequency component in the signal, and is calculated using the formula f. peak =argmax|X(f)|.

[0098] The frequency band energy ratio, α / β ratio, is used to measure the balance between relaxation and anxiety.

[0099] Time-frequency domain characteristics:

[0100] Combining time and frequency information, short-time fourier transform (STFT) and wavelet transform (WT) are used to perform time-frequency analysis on EEG. STFT generates a time-frequency representation, and the frequency distribution of the EEG signal at different times is extracted, defined as:

[0101]

[0102] Where w[nt] is the sliding window function, x[n] represents the sample points of the signal, S(t,f) represents the energy distribution at time t and frequency f, and e -j2πfn The kernel function represents the Fourier transform.

[0103] Wavelet transform performs multi-scale analysis of EEG signals by scaling and translating the mother wavelet function, extracting local frequency information of the signal, as shown below:

[0104]

[0105] Where psi(t) represents the mother wavelet function, s represents the scaling parameter, τ represents the time shift parameter, and ψ * This represents the complex conjugate of the mother wavelet.

[0106] Wavelet transform is used to capture the instantaneous spectral characteristics of emotional changes. Based on these characteristics, the instantaneous frequency of the signal at a specific moment is extracted to reflect the dynamic changes in the dominant frequency of the signal, defined as:

[0107]

[0108] Where phi(t) represents the phase of the signal, and the instantaneous frequency can reflect the frequency changes of physiological signals when emotions fluctuate rapidly.

[0109] After feature extraction is completed, in order to eliminate the differences in numerical range between different modal features, the system normalizes all modal features. The normalization formula is as follows:

[0110]

[0111] Where, x i ′ represents the original feature, x′ i,norm For the normalized features, min(x) i ′) and max(x i ′) represent the minimum and maximum values ​​of the modal feature, respectively.

[0112] Multimodal data fusion.

[0113] Building upon temporal alignment, linear transformation and contrastive learning strategies are used to align the feature-extracted multimodal data in both dimension and semantics, followed by weighted fusion through an attention mechanism. Specifically:

[0114] A fully connected layer is used to perform a linear transformation on the features of each modality, mapping each feature to the same vector feature space to achieve dimension alignment. The feature vector after the fully connected layer is represented as follows:

[0115] P i =W i F i +b i ;

[0116] Where i∈{EEG,HR,EDA,SpO2} represents the mode, W i Let b represent the weight matrix of the fully connected layer. i F represents the bias term. i Indicates modal feature input;

[0117] A contrastive learning framework is employed for semantic alignment of features from different modalities. The core objective of contrastive learning is to make features from different modalities under the same emotional state closer in the feature space, while simultaneously distinguishing features from different emotional states and distancing them from each other in the feature space. To achieve this goal, the cosine similarity of each pair of modal (i,j) features under the same emotional state is first calculated, using the following formula:

[0118]

[0119] Among them, P i k and Let i and j represent the feature projections of mode i and mode j, respectively, and k represent the same time sampling point;

[0120] Secondly, increase the feature similarity of positive sample pairs (modal feature pairs under the same emotional state) to bring these features closer together in the feature space, achieving semantic alignment. Reduce the feature similarity of negative sample pairs (modal feature pairs under different emotional states) to ensure the discriminative power of emotional states in the feature space.

[0121] To control for similarity differences, the contrastive learning loss between pairs of modes is calculated. The loss function for mode i and mode j is expressed as:

[0122]

[0123] In this model, the numerator represents the similarity of positive sample pairs under the same emotional state, while the denominator represents the similarity of negative sample pairs under different emotional states. By maximizing the numerator and minimizing the denominator, the system ensures that positive sample pairs have high similarity, while negative sample pairs have low similarity. τ represents a temperature parameter used to adjust the contrast of characteristic similarity. The temperature parameter can amplify or reduce the similarity difference between positive and negative samples, thereby enhancing the model's sensitivity to both positive and negative samples.

[0124] Finally, the final contrastive learning loss function is calculated. This final contrastive learning loss function is the sum of the losses between all mode pairs, expressed as:

[0125]

[0126] Where N is the total number of modalities (EEG, HR, EDA, SpO2, a total of 4). By minimizing the loss function, the system can simultaneously enhance the similarity of features of different modalities under the same emotional state and the discriminability of features under different emotional states, thereby improving the robustness of the emotion recognition system for emotion recognition tasks.

[0127] After modality alignment based on contrastive learning, a weighted fusion of features from different modalities is performed using an attention mechanism. The core idea of ​​the attention mechanism is to weight features according to their contribution, assigning different weights to each modality to generate a fused multimodal feature representation. First, the attention weight for each modality feature is calculated, expressed as:

[0128]

[0129] Among them, W α α is the learning parameter for attention weights. i P represents the attention weights for mode i. i P represents the projection of mode i. j This represents the projection of mode j.

[0130] Then, feature fusion is performed, and the fused feature vector F F Represented as:

[0131]

[0132] By using a weighted fusion mechanism based on attention, the system can effectively extract shared features from different modalities while preserving the contribution of each modality to emotion recognition.

[0133] Emotion recognition.

[0134] A sentiment classification model consisting of two fully connected layers identifies the fused multimodal features, classifying the user's emotional state y into positive or negative categories. The classification model maps the fused feature input to corresponding sentiment labels, using the following formula:

[0135] y = f classifier (F F ).

[0136] The implementation methods of the adaptive music intervention module include:

[0137] Construct a CVAE model, which includes a decoder and an encoder.

[0138] The data on emotional state, music features, and user preferences are preprocessed, and then the processed data is input into the CVAE model. Specifically:

[0139] Extract the current emotional state assessed by the current multimodal emotional state assessment module and represent it as an emotional distribution vector E. user Each emotion category (e.g., "pleasure" or "anxiety") corresponds to a probability value, reflecting the intensity and proportion of different emotions. Simultaneously, long-term music preference information is extracted from users' historical behavior data, including listening records, ratings, and skip rates, and represented as a personalized preference vector P. user Furthermore, feature information is extracted from each piece of music in the music library, including pitch, rhythm, melody, and emotional labels, and represented as music feature x, which is used to learn the distribution of different types of music in the latent space.

[0140] Information encoding.

[0141] The encoder receives various preprocessed data sets, maps the input data to the latent space using a neural network, generates the mean and variance of the latent variables, and outputs the distribution of the latent variables based on the latent variables. Specifically:

[0142] The encoder receives musical features x and user conditional input (emotional distribution vector E). user and preference vector P user The neural network maps input data to a latent space, generates the mean and variance of latent variables, and outputs the distribution of latent variables based on the latent variables, including:

[0143] The encoder maps input data to a latent space using a neural network, generating the mean μ of the latent variable z. φ (x,c) and variance Σ φ (x,c), where c=[E user ,P user [] indicates conditional input;

[0144] The encoder outputs the distribution of the latent variables based on the latent variables, which is represented as:

[0145]

[0146] To ensure the differentiability of the encoding process, a reparameterization technique is used to represent the latent variables as follows:

[0147] z = μ φ (x,c)+∈·Σ φ (x,c);

[0148] Where ∈ represents the standard normal distribution The model incorporates noise from the sampling process. By encoding music features and user conditions, the model can learn the relationship between music features and user emotions and historical preferences.

[0149] Information decoding.

[0150] The decoder takes latent variables and the user's emotional and behavioral conditional inputs to generate a new musical feature, specifically:

[0151] The decoder generates musical features that match the user's emotions and preferences by modeling latent variables and conditional inputs, represented as:

[0152]

[0153] Where, p θ (x′|z,c) represents the distribution of music features generated by the decoder, μ θ (z,c) and Σ θ (z,c) represent the mean and variance of the generated features, respectively;

[0154] The reconstruction loss and KL divergence loss functions are used to calculate the loss to ensure that the generated music feature x' meets the user's current emotional state and historical preferences.

[0155] The difference between the music features x' generated by the reconstruction loss metric and the original music features x is used to ensure that the decoder can generate high-quality music that meets the user's needs. This is expressed as:

[0156]

[0157] KL divergence loss is used to regularize the latent space, making the distribution of latent variables as close as possible to the standard normal distribution, thereby ensuring that the distribution of generated features is reasonable, expressed as:

[0158]

[0159] Where p(z) represents the standard normal distribution, D KLThis represents the KL divergence between music characteristics and the user's current emotional state and historical preference needs.

[0160] The total loss function is expressed as:

[0161]

[0162] Here, λ represents the hyperparameter that adjusts the reconstruction loss and the KL divergence weights. By minimizing the sum of these two losses, the model can progressively learn how to better recommend music that matches the user's current mood and historical behavior.

[0163] Music recommendation generation

[0164] The generated music features are matched with actual music features in a music library that pre-stores a large amount of music feature information. Based on similarity, the music is sorted to generate a personalized recommendation list that matches the user's emotional state. This personalized recommendation list is dynamically updated based on real-time changes in the user's emotional state and behavior. Specifically:

[0165] The music feature x' generated by calculating cosine similarity is compared with the music feature x of each song in the music library. music The similarity between them is calculated using the following formula:

[0166]

[0167] By calculating similarity, the system can find music that best matches the user's current mood and historical preferences. Based on similarity ranking, a personalized recommendation list is generated, typically selecting the top N songs most similar to the generated features. This ensures personalization while also considering the diversity of musical content. Subsequently, the personalized recommendation list is dynamically updated based on real-time changes in the user's emotional state and behavior. Through this dynamic update mechanism, the recommendation list maintains high relevance and freshness.

[0168] Taking a user with mild depressive symptoms as an example, the implementation process of the system is explained, and the specific process is as follows: Figure 4 As shown.

[0169] First, wear the monitoring device, and the system will intervene based on the monitoring signals.

[0170] Multimodal emotion state assessment module:

[0171] Step 1: Physiological signal acquisition.

[0172] Real-time monitoring of emotional state is achieved through wearing a multimodal physiological sensor device. The device includes the following sensors: 1) Portable three-lead EEG sensor: a three-channel EEG sensor (Fp1, Fp2, Fp2) with a sampling frequency of 250Hz, monitoring the user's brainwave activity in real time. 2) Smart bracelet integrated HR sensor: collects heart rate intervals once per second to monitor the user's heart rate variability. 3) Smart bracelet integrated EDA sensor: collects changes in the user's skin conductivity at a frequency of 1Hz to reflect the user's emotional arousal level. 4) Smart bracelet integrated SpO2 sensor: collects blood oxygen data once per second to monitor changes in the user's blood oxygen level.

[0173] Step 2: Signal preprocessing.

[0174] 1) The EEG signal is bandpass filtered (0.5Hz to 45Hz) to remove low-frequency interference (such as eye movement artifacts) and high-frequency noise (such as power supply noise). 2) The HR signal is smoothed to remove transient fluctuations in the heartbeat interval.

[0175] Step 3: Feature extraction.

[0176] The preprocessed physiological signals then proceed to the feature extraction stage. A unified time window is used to extract time-domain, frequency-domain, and time-frequency-domain features from EEG, HR, EDA, and SpO2 signals, respectively.

[0177] Step 4: Multimodal data fusion.

[0178] The system aligns the dimensions and emotional semantics of physiological signals from different modalities through a fusion module, and uses an attention mechanism to assign different weights to each modality, ultimately generating a fused feature vector.

[0179] Step 5: Emotional state recognition.

[0180] The fused feature vector is input into the emotion classification model. Through the neural network model, the system determines that the user's current emotion is "anxiety" (negative emotion) and passes this result to the adaptive music intervention module.

[0181] Adaptive music intervention module:

[0182] Based on the emotion recognition results, the system combines the user's music preference data to generate personalized music recommendations.

[0183] Step 1: Data Input and Encoding.

[0184] After acquiring the user's emotional state, the system encodes this state into an emotional distribution vector to capture and quantify the user's current emotional characteristics. Simultaneously, the system generates a personalized preference vector based on the user's historical music interaction data. Furthermore, the system generates a priori music feature vector by extracting multidimensional attributes (such as pitch, rhythm, volume, and emotional label) from each song in the music library.

[0185] Step 2: Application of the CVAE model.

[0186] Next, the system uses CVAE to process the conditional vectors (emotional distribution vector and preference vector) and music feature vectors. The CVAE encoder maps these features into the latent space, generating latent variables. Through reparameterization techniques, CVAE can generate music features that match the user's current emotional state.

[0187] Step 3: Music recommendation generation.

[0188] The system uses cosine similarity calculation to match the generated music feature vector with the music feature vectors in the music library. A higher cosine similarity indicates that the recommended music better matches the user's current mood. Ultimately, the system generates a personalized music recommendation list, primarily including light music and soothing tracks to help alleviate anxiety. The first few recommendations include: 1) a relaxing piece featuring slow piano and gentle strings; 2) a meditative piece with a soft rhythm; and 3) a soothing guitar performance that the user has frequently listened to in the past.

[0189] Step 4: Real-time feedback and dynamic updates.

[0190] The user begins listening to recommended music, and the system continues to monitor their physiological signals. If the user's anxiety subsides, the system dynamically adjusts the recommended music content, generating a new recommendation list. This entire process forms a closed loop, with the system dynamically recommending music based on the user's real-time emotional state.

[0191] Experiment and effect verification

[0192] In the user's feedback, after approximately 30 minutes of music intervention, her emotional state significantly improved, her anxiety decreased markedly, and her HR and EDA signals showed a more stable trend. The experimental results demonstrate that the music recommended by the system successfully alleviated her negative emotions, proving the effectiveness of the invention.

[0193] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multi-modal physiology feedback based adaptive music intervention system, characterized in that: The method comprises a multi-modal emotional state evaluation module and an adaptive music intervention module, the multi-modal emotional state evaluation module collects multi-modal physiological information of a user, performs multi-scale feature extraction and multi-modal data fusion on the multi-modal data, and evaluates an emotional state according to the fused data; the adaptive music intervention module generates music features matching a current emotion according to the evaluated emotional state, music features, and user preference data, and selects music to intervene in the user according to the music features; The multi-modal emotional state evaluation module comprises the following steps: Real-time collection of multi-modal physiological signals of a user through a wearable prefrontal three-lead electroencephalogram device and a smart bracelet, including electroencephalogram, heart rate, skin electricity, and blood oxygen saturation signals; Preprocessing of the original signals of the collected multi-modal data; Uniform time window of the multi-modal data, feature extraction of the preprocessed multi-modal data signals by using a mixed domain feature extraction strategy, and normalization processing of all modal features after the feature extraction; Dimension and semantic alignment of the multi-modal data after the feature extraction by using a linear transformation and a contrast learning strategy, and weighted fusion by using an attention mechanism; Emotion classification by using an emotion classification model composed of two full connection layers, and classification of the emotional state into positive or negative categories; The adaptive music intervention module comprises the following steps: Construction of a CVAE model, the CVAE model comprising a decoder and an encoder; Preprocessing of the emotional state, music features, and user preference data, and input of the processed data into the CVAE model; Reception of the preprocessed multi-modal data by the encoder, mapping of the input data to a latent space by using a neural network, generation of a mean value and a variance of a hidden variable, and output of a distribution of the latent variable according to the hidden variable; Input of the latent variable and the user's emotional and behavioral conditions into the decoder, and generation of a new music feature; Matching of the generated music feature with actual music features in a music library by using a similarity retrieval, generation of a personalized recommendation list according to a similarity ranking, and dynamic updating of the personalized recommendation list according to real-time changes in the user's emotional state and behavior.

2. The adaptive music intervention system based on multi-modal physiological feedback according to claim 1, characterized in that, The feature extraction strategy of the mixed domain comprises the following steps: Time domain feature extraction: extraction of the mean value, variance, and extreme value of the electroencephalogram, heart rate, skin electricity, and blood oxygen saturation signals, and extraction of the heart rate variability feature of the heart rate signal; Frequency domain feature extraction: conversion of the electroencephalogram signal from the time domain to the frequency domain by using the Fourier transform, and the frequency domain features including frequency domain energy, main frequency, and frequency domain energy ratio; Time-frequency domain feature extraction: combination of time and frequency information, time-frequency analysis of the electroencephalogram signal by using the short-time Fourier transform and the wavelet transform, generation of a time-frequency representation by using the short-time Fourier transform, extraction of the frequency distribution of the electroencephalogram signal at different times, multi-scale analysis of the electroencephalogram signal by using the wavelet transform, extraction of the local frequency information of the signal, and capture of the instantaneous frequency spectrum characteristics during the emotional change, and extraction of the instantaneous frequency of the signal at a certain time according to the instantaneous frequency spectrum characteristics.

3. The adaptive music intervention system based on multi-modal physiological feedback according to claim 2, characterized in that, The using linear transformation and contrast learning strategy to complete the alignment of the multi-modal data after feature extraction in dimension and semantics includes: The multi-modal feature is linearly transformed by using a fully connected layer, and the multi-modal feature is mapped to the same vector feature space, and the feature vector after the fully connected layer is represented as: ； wherein, denotes a modality, denotes a weight matrix of a fully connected layer, denotes a bias term, denotes a modality feature input; A contrastive learning framework is used to perform semantic alignment of features from different modalities. First, the semantic alignment of each pair of modalities under the same emotional state is calculated. The cosine similarity of features is calculated using the following formula: ； wherein, and denote the feature projections of the modalities and modalities respectively, denote the same time sample point; Secondly, the feature similarity of the positive sample pair is increased to ensure semantic alignment, and the feature similarity of the negative sample pair is reduced to ensure the distinguishability of the emotional state in the feature space, wherein the positive sample pair is a pair of modal features under the same emotional state, and the negative sample pair is a pair of modal features under different emotional states; Then, the contrastive learning loss between each pair of modalities is calculated, for modality and modality The loss function is represented as: ； wherein the numerator represents the similarity of the positive sample pair in the same emotion state, and the denominator represents the similarity of the negative sample pair in different emotion states, represents a temperature parameter for adjusting the contrast of the characteristic similarity, enhancing the sensitivity of the model to the positive and negative samples; Finally, the final contrast learning loss function is calculated, and the final contrast learning loss function is the sum of the losses between all modal pairs, represented as: ； wherein is the total number of modes.

4. The adaptive music intervention system based on multi-modal physiological feedback as claimed in claim 3, wherein, The weighted fusion by using the attention mechanism includes: First, the attention weight of each modal feature is calculated, represented as: ； wherein, is a learned parameter of an attention weight, is an attention weight for a modality is an attention weight for a modality denotes a projection of a modality denotes a projection of a modality denotes a projection of a modality denotes a projection of a modality Then, feature fusion is performed, and the fused feature vector is expressed as: 。 5. The adaptive music intervention system based on multi-modal physiological feedback as claimed in claim 1, wherein, The preprocessing of the emotional state, music feature and user preference data includes: extract the current emotional state evaluated by the current multi-modal emotional state evaluation module and represent it as an emotional distribution vector where each emotional category corresponds to a probability value; extract long-term music preference information in user historical behavior data, including user listening records, ratings and skip rates, and represent it as a user personalized preference vector ; extract feature information in each music in the music library, including pitch, rhythm, pitch, melody and emotional label, and represent it as music features x .

6. The adaptive music intervention system based on multi-modal physiological feedback as claimed in claim 5, wherein, The input data is mapped to the latent space by using the neural network, the mean and variance of the latent variable are generated, and the distribution of the latent variable is output according to the latent variable includes: An encoder maps input data to a latent space through a neural network, generating latent variables the mean and variance where denotes the conditional input; The encoder outputs the distribution of the latent variable according to the latent variable, represented as: ； The latent variable is represented by using the reparameterization trick, represented as: ； wherein, denotes noise sampled from a standard normal distribution denotes noise sampled from a standard normal distribution 7. The adaptive music intervention system based on multi-modal physiological feedback as claimed in claim 6, wherein, The decoder inputs the latent variable and the emotional and behavioral conditions of the user to generate a new music feature includes: The decoder generates the music feature matched with the user's emotion and preference by modeling the latent variable and the conditional input, represented as: ； wherein, denotes the music feature distribution generated by the decoder, and denote the mean and variance of the generated features, respectively; Loss is calculated using a reconstruction loss and a KL divergence loss function to ensure generated music features Meets the needs of the user's current emotional state and historical preferences; The reconstruction loss metric is represented as: ； The KL divergence loss function is represented as: ； wherein, denotes the standard normal distribution, denotes the KL divergence between the music features and the current emotional state of the user and the historical preference needs. The total loss function is represented as: ； where, denotes a hyperparameter that regulates the reconstruction loss versus the KL divergence weight.

Images