Self-adaptive music intervention system based on multi-modal physiological feedback

Abstract

The invention discloses a self-adaptive music intervention system based on multi-modal physiological feedback, and belongs to the technical field of monitoring and intervention of emotional states, the self-adaptive music intervention system comprises a multi-modal emotional state evaluation module and a self-adaptive music intervention module, the multi-modal emotional state evaluation module acquires multi-modal physiological information of a user, and the self-adaptive music intervention module performs music intervention on the multi-modal emotional state evaluation module; feature extraction and fusion are carried out on the multi-modal data, and the emotional state is evaluated according to the fused data; and the adaptive music intervention module generates music features matched with the current emotion according to the evaluated emotion state, the music features and preference data of the user, and selects music according to the music features to intervene the user. By the adoption of the method, the wearable forehead lobe three-lead electroencephalogram equipment and the intelligent bracelet are combined, physiological signals of the central nervous system and the autonomic nervous system are collected and analyzed in real time, music intervention is automatically adjusted, a personalized emotion adjusting scheme free of side effects is provided, and a closed-loop mental state regulation and control mechanism is formed.

Description

Technical Field

[0001] The present invention relates to the technical field of monitoring and intervening in emotional states, and in particular to an adaptive music intervention system based on multimodal physiological feedback. Background Art

[0002] With the acceleration of global economic pressure and pace of life, psychological disorders such as depression are becoming increasingly serious. Globally, there are about 280 million people suffering from depression, which has become one of the major public health issues affecting global mental health. It is worth noting that the onset of depression is showing a trend of younger age, and more and more teenagers are beginning to suffer from depression, further exacerbating the severity of this social problem.

[0003] Patients with depressive disorders usually show increased negative emotions and decreased positive emotions, which is closely related to their lack of emotion regulation ability. Studies have shown that depressed patients find it difficult to adopt effective emotion regulation strategies when facing emotional difficulties. Neuroscience research has found that when depressed patients process emotional stimuli, the responses of brain regions such as the amygdala and prefrontal cortex are different from those of healthy individuals. These brain regions play a key role in emotion regulation, and their dysfunction may be directly related to difficulties in emotion regulation.

[0004] The process of emotion regulation involves complex physiological and psychological mechanisms. Research in cognitive science and psychophysiology shows that the perception and regulation of emotions involve the joint action of the central nervous system (CNS) and the peripheral nervous system (PNS). When the external environment changes or stimuli appear, the sensory organs of the PNS quickly transmit information to the different levels of the brain center, activating the emotion perception and regulation mechanism. Brain structures closely related to emotional responses, such as the hypothalamus, amygdala and prefrontal cortex, integrate this information to form emotional experience, and then trigger the body's emotional expression through efferent neural pathways. This emotion regulation process not only activates the central nervous system of the brain, but also triggers functional changes in the PNS, especially the autonomic nervous system (ANS). The ANS includes the sympathetic nervous system (SNS) and the parasympathetic nervous system (PNS), and their activities can affect many aspects of physiological functions such as heart beat, breathing, blood pressure, digestion and metabolism. Therefore, obtaining central nervous system physiological signals (such as EEG) and autonomic nervous system physiological signals (such as electrocardiogram, skin electricity, etc.) during the emotion regulation process provides an objective basis for analyzing emotional state.

[0005] A large number of studies have shown that music therapy has a good therapeutic effect in the management of emotional disorders, especially depression. By regulating the response of the autonomic nervous system, music can effectively improve an individual's emotional state and enhance their mental health.

[0006] Therefore, there is an urgent need to develop an adaptive music intervention system based on multimodal physiological feedback to monitor and intervene in the emotional state of patients with depressive disorders. Summary of the invention

[0007] The purpose of the present invention is to provide an adaptive music intervention system based on multimodal physiological feedback, which combines a wearable prefrontal three-lead EEG device with a smart bracelet to collect and analyze physiological signals of the central nervous system and autonomic nervous system in real time, automatically adjust the music intervention, provide a side-effect-free, personalized emotion regulation solution, and form a closed-loop mental state regulation mechanism.

[0008] To achieve the above-mentioned objectives, the present invention provides an adaptive music intervention system based on multimodal physiological feedback, comprising a multimodal emotional state assessment module and an adaptive music intervention module. The multimodal emotional state assessment module collects multimodal physiological information of the user, performs multi-scale feature extraction and multimodal data fusion on the multimodal data, and assesses the emotional state according to the fused data; the adaptive music intervention module generates music features that match the current emotion according to the assessed emotional state, music features and user preference data, and selects music to intervene in the user according to the music features.

[0009] Preferably, the multimodal emotional state assessment module implementation method includes:

[0010] Through the wearable three-lead frontal EEG device and smart bracelet, the user's multimodal physiological signals are collected in real time, including EEG, heart rate, skin electricity and blood oxygen saturation signals;

[0011] Preprocessing the original signals of the collected multimodal data;

[0012] Unify the multimodal data time window, adopt the feature extraction strategy of the mixed domain, extract features from the preprocessed multimodal data signal, and normalize all modal features after feature extraction;

[0013] Use linear transformation and contrastive learning strategies to align the multimodal data after feature extraction in terms of dimension and semantics, and perform weighted fusion through the attention mechanism;

[0014] The emotion classification model composed of two fully connected layers is used to identify the fused multimodal features and classify the emotional state into positive or negative categories.

[0015] Preferably, the feature extraction strategy of the hybrid domain is adopted to extract features from the preprocessed multimodal data signal, including:

[0016] Time domain feature extraction: first extract the mean, variance and extreme value of EEG, heart rate, skin electricity and blood oxygen saturation signals, and then extract the heart rate variability characteristics of the heart rate signal;

[0017] Frequency domain feature extraction: Fourier transform is used to transform EEG signals from time domain into frequency domain features. Frequency domain features include frequency domain energy, main frequency and frequency domain energy ratio.

[0018] Time-frequency domain feature extraction: Combining time and frequency information, short-time Fourier transform and wavelet transform are used to perform time-frequency analysis on EEG signals. Short-time Fourier transform generates time-frequency representation and extracts the frequency distribution of EEG signals at different times. Wavelet transform performs multi-scale analysis on EEG signals by scaling and translating the mother wavelet function, extracts the local frequency information of the signal, and captures the instantaneous spectrum characteristics when emotions change. Based on the instantaneous spectrum characteristics, the instantaneous frequency of the signal at a certain moment is extracted.

[0019] Preferably, the method of using linear transformation and contrastive learning strategy to align the multimodal data after feature extraction in terms of dimension and semantics includes:

[0020] The fully connected layer is used to perform linear transformation on each modal feature and map each modal feature to the same vector feature space. The feature vector after passing through the fully connected layer is expressed as:

[0021] P i =W i F i +b i ;

[0022] Where i∈{EEG,HR,EDA,SpO2} represents the modality, W i represents the weight matrix of the fully connected layer, b i represents the bias term, F i represents the modal feature input;

[0023] The contrastive learning framework is used to perform semantic alignment of different modal features. First, the cosine similarity of each pair of modal (i, j) features under the same emotional state is calculated. The formula is:

[0024]

[0025] Among them, P i k and They represent the feature projections of mode i and mode j respectively, and k represents the same time sampling point;

[0026] Secondly, the feature similarity of the positive sample pairs is increased to ensure semantic alignment, and the feature similarity of the negative sample pairs is reduced to ensure the discrimination of emotional states in the feature space. The positive sample pairs are modal feature pairs under the same emotional state, and the negative sample pairs are modal feature pairs under different emotional states.

[0027] Then, the contrastive learning loss between the two modalities is calculated. The loss function for modality i and modality j is expressed as:

[0028]

[0029] Among them, the numerator represents the similarity of positive sample pairs under the same emotional state, the denominator represents the similarity of negative sample pairs under different emotional states, and τ represents the temperature parameter, which is used to adjust the contrast of feature similarity and enhance the sensitivity of the model to positive and negative samples;

[0030] Finally, the final contrastive learning loss function is calculated. The final contrastive learning loss function is the sum of the losses between all modality pairs, expressed as:

[0031]

[0032] Where N is the total number of modes.

[0033] Preferably, the weighted fusion of features of different modalities using the attention mechanism includes:

[0034] First, the attention weight of each modality feature is calculated, expressed as:

[0035]

[0036] Among them, W α is the learning parameter of the attention weight, α i is the attention weight of modality i, P i represents the projection of mode i, P j represents the projection of mode j;

[0037] Then, feature fusion is performed, and the fused feature vector F F It is expressed as:

[0038]

[0039] Preferably, the method for implementing the adaptive music intervention module includes:

[0040] Build a CVAE model, which includes a decoder and an encoder;

[0041] Preprocess the emotional state, music features, and user preference data, and input the processed data into the CVAE model;

[0042] The encoder receives a variety of preprocessed data, maps the input data to the latent space through a neural network, generates the mean and variance of the latent variables, and outputs the distribution of the latent variables based on the latent variables;

[0043] The decoder inputs the latent variables with the user’s emotional and behavioral conditions to generate a new music feature;

[0044] The generated music features are matched with the actual music features in the music library through similarity retrieval, sorted according to similarity, and a personalized recommendation list that matches the user's emotional state is generated. The personalized recommendation list is dynamically updated according to real-time changes in the user's emotional state and behavior.

[0045] Preferably, the data preprocessing includes:

[0046] Extract the current emotional state evaluated by the current multimodal emotional state evaluation module and represent it as the emotional distribution vector E user , where each emotion category corresponds to a probability value; extract the long-term music preference information in the user's historical behavior data, including the user's listening record, rating, and skip rate, and express it as the user's personalized preference vector P user ; Extract feature information from each piece of music in the music library, including tone, rhythm, pitch, melody, and emotional labels, and represent it as music feature x.

[0047] Preferably, mapping the input data to the latent space through a neural network, generating the mean and variance of the latent variables, and outputting the distribution of the latent variables according to the latent variables comprises:

[0048] The encoder maps the input data to the latent space through a neural network and generates the mean μ of the latent variable z. φ (x,c) and variance Σ φ (x,c), where c = [E user ,P user ] indicates conditional input;

[0049] The encoder outputs the distribution of latent variables based on the latent variables, expressed as:

[0050]

[0051] Using the reparameterization technique, the latent variables are represented as:

[0052] z=μ φ (x,c)+∈·Σ φ (x,c);

[0053] Among them, ∈ represents the standard normal distribution The noise sampled in .

[0054] Preferably, the decoder inputs the latent variables and the user's emotions and behavior conditions to generate a new music feature including:

[0055] The decoder generates music features that match the user's emotions and preferences by modeling the latent variables and conditional inputs, expressed as:

[0056]

[0057] Among them, p θ (x′|z,c) represents the music feature distribution generated by the decoder, μ θ (z,c) and Σ θ (z,c) represent the mean and variance of the generated features respectively;

[0058] The loss is calculated using reconstruction loss and KL divergence loss functions to ensure that the generated music feature x' meets the user's current emotional state and historical preferences.

[0059] The reconstruction loss metric is expressed as:

[0060]

[0061] The KL divergence loss function is expressed as:

[0062]

[0063] Among them, p(z) represents the standard normal distribution, D KL Represents the KL divergence between music features and the user's current emotional state and historical preference needs;

[0064] The total loss function is expressed as:

[0065]

[0066] Among them, λ represents the hyperparameter for adjusting the reconstruction loss and KL divergence weight.

[0067] Therefore, the present invention adopts the above-mentioned adaptive music intervention system based on multimodal physiological feedback, which has the following beneficial effects:

[0068] (1) Combining a wearable three-lead prefrontal EEG acquisition device and a smart bracelet, it can monitor the activity of the prefrontal emotional response area of ​​the brain in real time to obtain heart rate, blood oxygen and other signals, and conduct a comprehensive assessment of the user's emotional state;

[0069] (2) By integrating multimodal physiological data such as electroencephalogram, heart rate, skin electricity, and blood oxygen saturation, and combining it with the modal feature alignment method of contrastive learning, high-precision emotional state recognition is achieved;

[0070] (3) Through the weighted fusion of the attention mechanism, the contribution of each modality to emotion recognition is dynamically adjusted, ensuring the robustness and flexibility of the system under complex emotional states;

[0071] (4) The adaptive music intervention module based on CVAE can generate personalized music recommendations based on the user’s current emotional state and historical behavior data, thereby effectively helping users regulate their emotions and improving user experience and system intelligence.

[0072] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 is a system architecture diagram of an embodiment of the present invention;

[0074] Figure 2 is a detailed flow chart of a multimodal emotional state assessment module according to an embodiment of the present invention;

[0075] Figure 3 A detailed flow chart of an adaptive music intervention module according to an embodiment of the present invention;

[0076] Figure 4 The following is a flowchart of a system implementation of an embodiment of the present invention. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0078] Example

[0079] Reference Figure 1-3 The present invention provides an adaptive music intervention system based on multimodal physiological feedback, including a multimodal emotional state assessment module and an adaptive music intervention module.

[0080] The multimodal emotional state assessment module collects the user's multimodal physiological information, extracts multi-scale features and fuses multimodal data, and assesses the emotional state based on the fused data. The adaptive music intervention module generates music features that match the current emotion based on the assessed emotional state, music features and user preference data, and selects music to intervene in the user based on the music features.

[0081] The implementation method of the multimodal emotional state assessment module includes:

[0082] Physiological signal acquisition.

[0083] Through the wearable three-lead prefrontal EEG device and smart bracelet, the user's multimodal physiological signals are collected in real time, including electroencephalogram (EEG), heart rate (HR), electrodermal activity (EDA) and blood oxygen saturation (SpO2) signals. The wearable three-lead prefrontal EEG device includes 3 channels (Fp1, Fpz and Fp2). EEG signals are collected at a sampling frequency of 250Hz through a 3-channel non-invasive sensor, and HR, EDA and SpO2 signals are collected at a frequency of 1Hz through the built-in integrated sensor of the smart bracelet. These signals can reflect the user's physiological changes and thus reveal emotional fluctuations. The collected physiological signals are set as follows:

[0084]

[0085] Among them, x EEG ,x HR ,x EDA , They represent physiological signals of different modalities.

[0086] Preprocess the original signals of the collected multimodal data.

[0087] The collected raw signals may contain noise and artifacts, so these signals are preprocessed. EEG signals are bandpass filtered (0.5Hz to 45Hz) to remove low-frequency interference (such as eye movement artifacts) and high-frequency noise (such as power supply noise). HR signals are smoothed and filtered to remove instantaneous fluctuations in heartbeat intervals.

[0088] Feature extraction.

[0089] Unify the multimodal data time window to ensure that all modal signals are synchronized in the same time window to achieve time domain alignment of multimodal data signals. Adopt the hybrid domain feature extraction strategy to extract features from the preprocessed multimodal data signal and normalize all modal features after feature extraction. Specifically include:

[0090] The physiological signals of different modes are divided into w Alignment is performed to ensure that all modal signals are synchronized in the same time window. For high-frequency signals (EEG), each T w For low-frequency signals (such as HR, EDA, and SpO2), T w Statistical processing of multi-second data.

[0091] Extracting multimodal data features includes extracting time domain, frequency domain and time-frequency domain features of user EEG, HR, EDA and SpO2 signals.

[0092] Time domain characteristics:

[0093] In the time domain, statistical features that reflect the fluctuation characteristics and overall trends of the signal are directly extracted from the time series of the signal. Time domain features help capture the overall changes of physiological signals during emotional fluctuations. Specifically, the mean, variance, and extreme values ​​of the electroencephalogram, heart rate, skin electricity, and blood oxygen saturation signals are extracted, and then the heart rate variability (HRV) features of the heart rate signal are further extracted.

[0094] Frequency domain characteristics:

[0095] Use Fourier transform to transform EEG signals from time domain to frequency domain features and extract energy features of different frequency bands. Frequency domain features include:

[0096] Frequency domain energy E f , the energy of a specific frequency band (the α, β, and θ bands are associated with emotions such as relaxation, tension, and alertness, respectively), expressed as Where X(f) represents the Fourier transform result of the signal, and f1 and f2 represent the start and end frequencies of the frequency band.

[0097] Main frequency f peak , is the frequency component with the largest signal energy, indicating the most important frequency component in the signal. The calculation formula is f peak =argmax|X(f)|.

[0098] The frequency band energy ratio, α / β ratio, is used to measure the balance between relaxation and anxiety states.

[0099] Time-frequency domain features:

[0100] Combining time and frequency information, short-time Fourier transform (STFT) and wavelet transform (WT) are used to perform time-frequency analysis on EEG. STFT generates time-frequency representation and extracts the frequency distribution of EEG signals at different times, which is defined as:

[0101]

[0102] Where w[nt] is the sliding window function, x[n] represents the sample point of the signal, S(t,f) represents the energy distribution at time t and frequency f, and e -j2πfn Represents the kernel function of the Fourier transform.

[0103] Wavelet transform performs multi-scale analysis on EEG signals by scaling and translating the mother wavelet function to extract the local frequency information of the signal, which can be expressed as:

[0104]

[0105] Among them, psi(t) represents the mother wavelet function, s represents the scale parameter, τ represents the time translation parameter, ψ * represents the complex conjugate of the mother wavelet.

[0106] The instantaneous spectrum characteristics of emotion changes are captured by wavelet transform. According to the instantaneous spectrum characteristics, the instantaneous frequency of the signal at a certain moment is extracted to reflect the dynamic changes of the dominant frequency of the signal, which is defined as:

[0107]

[0108] Among them, phi(t) represents the phase of the signal, and the instantaneous frequency can reflect the frequency change of the physiological signal when the emotion fluctuates rapidly.

[0109] After feature extraction is completed, in order to eliminate the differences in the numerical ranges of different modal features, the system normalizes all modal features. The normalization formula is as follows:

[0110]

[0111] Among them, x i ′ is the original feature, x′ i,norm is the normalized feature, min(x i ′) and max(x i ′) are the minimum and maximum values ​​of the modal characteristics, respectively.

[0112] Multimodal data fusion.

[0113] On the basis of time domain alignment, linear transformation and contrastive learning strategies are used to align the multimodal data after feature extraction in terms of dimension and semantics, and weighted fusion is performed through the attention mechanism. Specifically:

[0114] A fully connected layer is used to perform linear transformation on each modal feature and map each modal feature to the same vector feature space to achieve dimensional alignment. The feature vector after passing through the fully connected layer is expressed as:

[0115] P i =W i F i +b i ;

[0116] Where i∈{EEG,HR,EDA,SpO2} represents the modality, W i represents the weight matrix of the fully connected layer, b i represents the bias term, F i represents the modal feature input;

[0117] The contrastive learning framework is used to align the semantics of different modal features. The core goal of contrastive learning is to make different modal features under the same emotional state closer in the feature space by comparing them; at the same time, to distinguish the modal features under different emotional states and make them farther away from each other in the feature space. To achieve this goal, the cosine similarity of each pair of modal (i, j) features under the same emotional state is first calculated, and the formula is:

[0118]

[0119] Among them, P i k and They represent the feature projections of mode i and mode j respectively, and k represents the same time sampling point;

[0120] Secondly, we increase the feature similarity of positive sample pairs (modal feature pairs under the same emotional state) and bring these features closer in the feature space to achieve semantic alignment. We also reduce the feature similarity of negative sample pairs (modal feature pairs under different emotional states) to ensure the discrimination of emotional states in the feature space.

[0121] In order to control the similarity difference, the contrastive learning loss between the two modalities is calculated. The loss function for modality i and modality j is expressed as:

[0122]

[0123] Among them, the numerator represents the similarity of positive sample pairs under the same emotional state, and the denominator represents the similarity of negative sample pairs under different emotional states. By maximizing the numerator and minimizing the denominator, the system can ensure that the similarity of positive sample pairs is high, while the similarity of negative sample pairs is low. τ represents the temperature parameter, which is used to adjust the contrast of feature similarity. Through the temperature parameter, the similarity difference between positive and negative samples can be magnified or reduced, thereby enhancing the sensitivity of the model to positive and negative samples.

[0124] Finally, the final contrastive learning loss function is calculated. The final contrastive learning loss function is the sum of the losses between all modality pairs, expressed as:

[0125]

[0126] Where N is the total number of modalities (EEG, HR, EDA, SpO2, a total of 4). By minimizing the loss function, the system can simultaneously enhance the similarity of different modal features under the same emotional state and the distinguishability of features under different emotional states, thereby improving the robustness of the emotion recognition system for emotion recognition tasks.

[0127] After modal alignment based on contrastive learning, the attention mechanism is used to perform weighted fusion of features of different modalities. The core idea of ​​the attention mechanism is to weight them according to the contribution of different modalities and assign different weights to each modality, thereby generating a fused multimodal feature representation. First, the attention weight of each modal feature is calculated, expressed as:

[0128]

[0129] Among them, W α is the learning parameter of the attention weight, α i is the attention weight of modality i, P i represents the projection of mode i, P j represents the projection of mode j.

[0130] Then, feature fusion is performed, and the fused feature vector F F It is expressed as:

[0131]

[0132] Through the weighted fusion of the attention mechanism, the system can effectively extract shared features of different modalities while retaining the contribution of each modality to emotion recognition.

[0133] Emotion recognition.

[0134] The emotion classification model composed of two fully connected layers recognizes the fused multimodal features, identifies the user's emotional state y, and classifies the emotional state into positive or negative categories. The classification model maps the fused feature input to the corresponding emotion label, and the formula is:

[0135] y=f classifier (F F ).

[0136] The implementation method of the adaptive music intervention module includes:

[0137] Build a CVAE model, which includes a decoder and an encoder.

[0138] Preprocess the emotional state, music features and user preference data, and input the processed data into the CVAE model. Specifically:

[0139] Extract the current emotional state evaluated by the current multimodal emotional state evaluation module and represent it as the emotional distribution vector E user , where each emotion category (such as "happy" or "anxious") corresponds to a probability value, reflecting the intensity and proportion of different emotions. At the same time, the long-term music preference information in the user's historical behavior data is extracted, including the user's listening record, rating, and skip rate, and is represented as the user's personalized preference vector P user In addition, the characteristic information of each piece of music in the music library, including tone, rhythm, pitch, melody and emotional labels, is extracted and represented as music feature x, which is used to learn the distribution of different types of music in the latent space.

[0140] Information encoding.

[0141] The encoder receives a variety of preprocessed data, maps the input data to the latent space through a neural network, generates the mean and variance of the latent variables, and outputs the distribution of the latent variables based on the latent variables, specifically:

[0142] The encoder receives the music feature x and the user condition input (emotion distribution vector E user and the preference vector P user ), the input data is mapped to the latent space through the neural network, the mean and variance of the latent variables are generated, and the distribution of the latent variables output according to the latent variables includes:

[0143] The encoder maps the input data to the latent space through a neural network and generates the mean μ of the latent variable z. φ (x,c) and variance Σ φ (x,c), where c = [E user ,P user ] indicates conditional input;

[0144] The encoder outputs the distribution of latent variables based on the latent variables, expressed as:

[0145]

[0146] To ensure the differentiability of the encoding process, a reparameterization technique is used to represent the latent variables as:

[0147] z=μ φ (x,c)+∈·Σ φ (x,c);

[0148] Among them, ∈ represents the standard normal distribution By encoding music features and user conditions, the model can learn the relationship between music features and user emotions and historical preferences.

[0149] Information decoding.

[0150] The decoder inputs the latent variables with the user’s emotional and behavioral conditions to generate a new music feature, specifically:

[0151] The decoder generates music features that match the user's emotions and preferences by modeling the latent variables and conditional inputs, expressed as:

[0152]

[0153] Among them, p θ (x′|z,c) represents the music feature distribution generated by the decoder, μ θ (z,c) and Σ θ (z,c) represent the mean and variance of the generated features respectively;

[0154] The loss is calculated using reconstruction loss and KL divergence loss function to ensure that the generated music feature x' meets the user's current emotional state and historical preferences.

[0155] The reconstruction loss measures the difference between the generated music features x' and the original music features x, ensuring that the decoder can generate high-quality music that meets user needs, expressed as:

[0156]

[0157] KL divergence loss is used to regularize the latent space so that the distribution of latent variables is as close to the standard normal distribution as possible, thereby ensuring that the distribution of generated features is reasonable, expressed as:

[0158]

[0159] Among them, p(z) represents the standard normal distribution, D KLRepresents the KL divergence between music features and the user's current emotional state and historical preference needs;

[0160] The total loss function is expressed as:

[0161]

[0162] Among them, λ represents the hyperparameter that adjusts the reconstruction loss and KL divergence weight. By minimizing the sum of these two losses, the model can gradually learn how to better recommend music that matches the user's current mood and historical behavior.

[0163] Music recommendation generation

[0164] The generated music features are matched with the actual music features in the music library through similarity retrieval. The music library pre-stores a large amount of music feature information. According to the similarity sorting, a personalized recommendation list that meets the user's emotional state is generated, and the personalized recommendation list is dynamically updated according to the real-time changes in the user's emotional state and behavior. Specifically:

[0165] The music feature x' generated by cosine similarity calculation is compared with each music feature x in the music library music The similarity between them is calculated as:

[0166]

[0167] By calculating similarity, the system can find the music that best matches the user's current mood and historical preferences. It generates a personalized recommendation list based on similarity sorting, usually selecting the top N songs that are most similar to the generated features for recommendation. The recommendation list ensures personalization while also taking into account the diversity of music content. Afterwards, the personalized recommendation list is dynamically updated based on the real-time changes in the user's emotional state and behavior. Through this dynamic update mechanism, the recommendation list can always maintain high relevance and freshness.

[0168] Taking a user with mild depression as an example, the implementation process of the system is explained. The specific process is as follows: Figure 4 shown.

[0169] First, wear the monitoring device, and the system intervenes based on the monitoring signals.

[0170] Multimodal emotional state assessment module:

[0171] Step 1: Physiological signal collection.

[0172] Real-time monitoring of emotional state is performed by wearing a multimodal physiological sensor device. The device includes the following sensors: 1) Portable three-lead EEG sensor: three-channel EEG sensor (Fp1, Fpz, Fp2), with a sampling frequency of 250Hz, to monitor the user's brain wave activity in real time. 2) Smart bracelet integrated HR sensor: collects heartbeat intervals once a second to monitor the user's heart rate variability. 3) Smart bracelet integrated EDA sensor: collects changes in the user's skin conductivity at a frequency of 1Hz to reflect the user's emotional arousal level. 4) Smart bracelet integrated SpO2 sensor: collects blood oxygen data once a second to monitor changes in the user's blood oxygen level.

[0173] Step 2: Signal preprocessing.

[0174] 1) The EEG signal is band-pass filtered (0.5 Hz to 45 Hz) to remove low-frequency interference (such as eye movement artifacts) and high-frequency noise (such as power supply noise). 2) The HR signal is smoothed to remove instantaneous fluctuations in the heartbeat interval.

[0175] Step 3: Feature extraction.

[0176] After preprocessing, the physiological signals enter the feature extraction stage. The time window is unified to extract the time domain, frequency domain and time-frequency domain features of EEG, HR, EDA and SpO2 signals respectively.

[0177] Step 4: Multimodal data fusion.

[0178] The system aligns the dimensions and emotional semantics of physiological signals of different modalities through a fusion module, and uses an attention mechanism to assign different weights to each modality, ultimately generating a fused feature vector.

[0179] Step 5: Emotional state recognition.

[0180] The fused feature vector is input into the emotion classification model. Through the neural network model, the system determines that the user's current emotion is "anxiety" (negative emotion) and passes this result to the adaptive music intervention module.

[0181] Adaptive Music Intervention Modules:

[0182] Based on the emotion recognition results, the system combines the user's music preference data to generate personalized music recommendations for them.

[0183] Step 1: Data input and coding.

[0184] After obtaining the user's emotional state, the system encodes the emotional state into an emotional distribution vector to capture and quantify the user's current emotional characteristics. At the same time, the system generates a personalized preference vector based on the user's historical music interaction data. In addition, the system generates a priori music feature vector by extracting the multidimensional attributes of each piece of music in the music library (such as tone, rhythm, pitch, emotional tags, etc.).

[0185] Step 2: Application of CVAE model.

[0186] Next, the system uses CVAE to process the conditional vector (emotional distribution vector and preference vector) and the music feature vector. The CVAE encoder maps these features into the latent space to generate latent variables. Through the reparameterization technique, CVAE is able to generate music features that match the user's current emotional state.

[0187] Step 3: Generate music recommendations.

[0188] The system matches the generated music feature vector with the music feature vector in the music library through cosine similarity calculation. The higher the cosine similarity, the more the recommended music matches the user's current mood. Finally, the system generates a personalized music recommendation list, mainly including some light music and music with a soothing rhythm to help users relieve anxiety. The top few recommendations include: 1) a relaxing music with slow piano and soft strings; 2) a meditation music with a soft rhythm; 3) a soothing guitar performance that the user has listened to frequently in the past.

[0189] Step 4: Real-time feedback and dynamic updates.

[0190] The user starts listening to the recommended music, and the system continues to monitor his physiological signals. If the user's anxiety is relieved, the system will dynamically adjust the recommended music content and generate a new recommendation list. The whole process forms a closed loop, and the system makes dynamic music recommendations based on the user's real-time emotional state.

[0191] Experiment and effect verification

[0192] According to the user's feedback, after about 30 minutes of music intervention, her emotional state was significantly improved, her anxiety was significantly reduced, and her HR and EDA signals also showed a more stable trend. The experimental results show that the music recommended by the system successfully alleviated her negative emotions, proving the effectiveness of the present invention.

[0193] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. An adaptive music intervention system based on multimodal physiological feedback, characterized in that: The system comprises a multimodal emotional state assessment module and an adaptive music intervention module. The multimodal emotional state assessment module collects multimodal physiological information of the user, performs multi-scale feature extraction and multimodal data fusion on the multimodal data, and assesses the emotional state according to the fused data; the adaptive music intervention module generates music features that match the current emotion according to the assessed emotional state, music features and user preference data, and selects music to intervene in the user according to the music features.

2. The adaptive music intervention system based on multimodal physiological feedback according to claim 1, characterized in that: The multimodal emotional state assessment module implementation method includes: Through the wearable three-lead frontal EEG device and smart bracelet, the user's multimodal physiological signals are collected in real time, including EEG, heart rate, skin electricity and blood oxygen saturation signals; Preprocessing the original signals of the collected multimodal data; Unify the multimodal data time window, adopt the feature extraction strategy of the mixed domain, extract features from the preprocessed multimodal data signal, and normalize all modal features after feature extraction; Use linear transformation and contrastive learning strategies to align the multimodal data after feature extraction in terms of dimension and semantics, and perform weighted fusion through the attention mechanism; The emotion classification model composed of two fully connected layers is used to identify the fused multimodal features and classify the emotional state into positive or negative categories.

3. The adaptive music intervention system based on multimodal physiological feedback according to claim 2, characterized in that: The feature extraction strategy using the hybrid domain to extract features from the preprocessed multimodal data signal includes: Time domain feature extraction: first extract the mean, variance and extreme value of EEG, heart rate, skin electricity and blood oxygen saturation signals, and then extract the heart rate variability characteristics of the heart rate signal; Frequency domain feature extraction: Fourier transform is used to transform EEG signals from time domain into frequency domain features. Frequency domain features include frequency domain energy, main frequency and frequency domain energy ratio. Time-frequency domain feature extraction: Combining time and frequency information, short-time Fourier transform and wavelet transform are used to perform time-frequency analysis on EEG signals. Short-time Fourier transform generates time-frequency representation and extracts the frequency distribution of EEG signals at different times. Wavelet transform performs multi-scale analysis on EEG signals by scaling and translating the mother wavelet function, extracts the local frequency information of the signal, and captures the instantaneous spectrum characteristics when emotions change. Based on the instantaneous spectrum characteristics, the instantaneous frequency of the signal at a certain moment is extracted.

4. The adaptive music intervention system based on multimodal physiological feedback according to claim 3, characterized in that: The method of using linear transformation and contrastive learning strategy to align the multimodal data after feature extraction in terms of dimension and semantics includes: The fully connected layer is used to perform linear transformation on each modal feature and map each modal feature to the same vector feature space. The feature vector after passing through the fully connected layer is expressed as: P i =W i F i +b i ; Where i∈{EEG,HR,EDA,SpO2} represents the modality, W i represents the weight matrix of the fully connected layer, b i represents the bias term, F i represents the modal feature input; The contrastive learning framework is used to perform semantic alignment of different modal features. First, the cosine similarity of each pair of modal (i, j) features under the same emotional state is calculated. The formula is: in, and They represent the feature projections of mode i and mode j respectively, and k represents the same time sampling point; Secondly, the feature similarity of the positive sample pairs is increased to ensure semantic alignment, and the feature similarity of the negative sample pairs is reduced to ensure the discrimination of emotional states in the feature space. The positive sample pairs are modal feature pairs under the same emotional state, and the negative sample pairs are modal feature pairs under different emotional states. Then, the contrastive learning loss between the two modalities is calculated. The loss function for modality i and modality j is expressed as: Among them, the numerator represents the similarity of positive sample pairs under the same emotional state, the denominator represents the similarity of negative sample pairs under different emotional states, and τ represents the temperature parameter, which is used to adjust the contrast of feature similarity and enhance the sensitivity of the model to positive and negative samples; Finally, the final contrastive learning loss function is calculated. The final contrastive learning loss function is the sum of the losses between all modality pairs, expressed as: Where N is the total number of modes.

5. The adaptive music intervention system based on multimodal physiological feedback according to claim 4, characterized in that: The weighted fusion of features of different modalities using the attention mechanism includes: First, the attention weight of each modality feature is calculated, expressed as: Among them, W α is the learning parameter of the attention weight, α i is the attention weight of modality i, P i represents the projection of mode i, P j represents the projection of mode j; Then, feature fusion is performed, and the fused feature vector F F It is expressed as:

6. The adaptive music intervention system based on multimodal physiological feedback according to claim 5, characterized in that: The method for implementing the adaptive music intervention module includes: Build a CVAE model, which includes a decoder and an encoder; Preprocess the emotional state, music features, and user preference data, and input the processed data into the CVAE model; The encoder receives a variety of preprocessed data, maps the input data to the latent space through a neural network, generates the mean and variance of the latent variables, and outputs the distribution of the latent variables based on the latent variables; The decoder inputs the latent variables with the user’s emotional and behavioral conditions to generate a new music feature; The generated music features are matched with the actual music features in the music library through similarity retrieval, sorted according to similarity, and a personalized recommendation list that matches the user's emotional state is generated. The personalized recommendation list is dynamically updated according to real-time changes in the user's emotional state and behavior.

7. The adaptive music intervention system based on multimodal physiological feedback according to claim 6, characterized in that: The data preprocessing includes: Extract the current emotional state evaluated by the current multimodal emotional state evaluation module and represent it as the emotional distribution vector E user , where each emotion category corresponds to a probability value; extract the long-term music preference information in the user's historical behavior data, including the user's listening record, rating, and skip rate, and express it as the user's personalized preference vector P user ; Extract feature information from each piece of music in the music library, including tone, rhythm, pitch, melody, and emotional labels, and represent it as music feature x.

8. The adaptive music intervention system based on multimodal physiological feedback according to claim 7, characterized in that: The method of mapping the input data to the latent space through the neural network, generating the mean and variance of the latent variables, and outputting the distribution of the latent variables according to the latent variables includes: The encoder maps the input data to the latent space through a neural network and generates the mean μ of the latent variable z. φ (x,c) and variance Σ φ (x,c), where c = [E user ,P user ] indicates conditional input; The encoder outputs the distribution of latent variables based on the latent variables, expressed as: Using the reparameterization technique, the latent variables are represented as: z=μ φ (x,c)+∈·Σ φ (x,c); Among them, ∈ represents the standard normal distribution The noise sampled in .

9. The adaptive music intervention system based on multimodal physiological feedback according to claim 8, characterized in that: The decoder inputs the latent variables with the user's emotional and behavioral conditions to generate a new music feature including: The decoder generates music features that match the user's emotions and preferences by modeling the latent variables and conditional inputs, expressed as: Among them, p θ (x′|z,c) represents the music feature distribution generated by the decoder, μ θ (z,c) and Σ θ (z,c) represent the mean and variance of the generated features respectively; The loss is calculated using reconstruction loss and KL divergence loss functions to ensure that the generated music feature x' meets the user's current emotional state and historical preferences. The reconstruction loss metric is expressed as: The KL divergence loss function is expressed as: Among them, p(z) represents the standard normal distribution, D KL Represents the KL divergence between music features and the user's current emotional state and historical preference needs; The total loss function is expressed as: Among them, λ represents the hyperparameter for adjusting the reconstruction loss and KL divergence weight.

Images