Wearable emotion perception automatic composition equipment and method based on electroencephalogram signals

Abstract

The invention discloses wearable emotion perception automatic composition equipment and method based on electroencephalogram signals, and belongs to the crossing field of wearable technology and music generation. The equipment comprises a wearable electroencephalogram acquisition module, an emotion recognition module, an automatic composition module, a music playing module and a main control module, and is characterized in that real-time electroencephalogram signals of a user are acquired, power spectral density characteristics of delta, theta, alpha, beta and gamma waves are extracted, an emotion state is recognized by utilizing a four-classification model; and the LSTM network and the genetic algorithm are combined to dynamically generate music matched with the current emotion. Real-time linkage of emotion perception and music generation is achieved, the equipment is high in wearable performance, the generated music has individuation and emotion adaptability, and the method is suitable for scenes such as emotion adjustment and music treatment.

Description

Technical Field

[0001] This invention relates to the fields of wearable devices, brain-computer interfaces, and automatic music composition, and particularly to a wearable device and method for dynamically generating appropriate music based on real-time perception of user emotions using electroencephalogram (EEG) signals. Background Technology

[0002] With increasing societal pressures, the demand for emotion management and music therapy is growing. Current technologies primarily rely on automated music composition systems that generate music based on fixed rules or music datasets, lacking integration with the user's real-time state. While EEG emotion sensing devices can identify emotions, they are mostly used for monitoring and not deeply integrated with music generation.

[0003] Existing technologies have significant limitations in meeting the needs for personalized emotion management and music experiences. Traditional automatic music composition methods, such as systems based on LSTM networks and genetic algorithms, can generate music, but they do not take into account the user's real-time state, resulting in content lacking emotional adaptability. Music therapy relies on experience, and due to significant differences in individual emotional perception, it is difficult to accurately assess its effectiveness. Furthermore, mainstream emotion recognition technologies are based on behavioral or physiological data and cannot reflect internal emotional fluctuations in a timely manner.

[0004] Electroencephalogram (EEG) signals offer the possibility of accurate emotion recognition, as they contain rich information about psychological states. Wearable EEG devices can acquire EEG signals in real time, and the power spectral density of each frequency band can be extracted using the Welch method, combined with a four-class classification model to achieve emotion classification. However, existing EEG devices are mostly bulky and expensive. For example, acquisition systems based on the ADS1299, while stable in performance, are limited in popularity due to price and size constraints, and lack integration with automatic music composition systems, making it difficult to achieve real-time linkage between emotions and music.

[0005] This invention aims to solve the above problems by integrating EEG emotion perception and dynamic music generation technology to provide a wearable, real-time responsive, and personalized automatic music composition solution. Summary of the Invention

[0006] The present invention aims to provide a wearable emotion-sensing automatic music composition device and method based on electroencephalogram (EEG) signals, in order to solve the problems of disconnect between automatic music composition and user emotions and the limited functionality of EEG emotion sensing devices in the prior art, and to realize real-time linkage between emotions and music and personalized music generation.

[0007] This invention consists of five main modules:

[0008] 1. Wearable EEG acquisition module, used to collect EEG signals and use the acquired EEG signals as the signal source for the emotion recognition module. It adopts the ADS1299 chip, 8-channel acquisition, pre-filter RC filter (cutoff frequency 100Hz), electrodes integrated into the headset, lightweight and comfortable to wear.

[0009] 2. Emotion Recognition Module: This module performs emotion recognition on the acquired EEG signals. It utilizes the on-chip TinyML framework on the STM32H7 to calculate the EEG Welch power spectrum and wavelet entropy in real-time at 16ms intervals, extracting the δ-γ five-band energy and simultaneously analyzing the microstate sequence and Hjorth parameters. These are then fused into a 64-dimensional emotion-cognition vector z_emo-cog, achieving high-frequency and accurate recognition of emotional states.

[0010] 3. Automatic composition module, used to compose action music based on emotional information analyzed from EEG signals. This part uses a two-layer LSTM to predict the next pitch or time value based on the real-time emotion vector z_emo-cog, and is optimized by an online genetic algorithm with 10 individuals: fitness is integrated in real time with 2-gram, 3-gram and music rule scores, generating 4 bars of MIDI per second, while caching the next 4 bars, to achieve low-latency personalized automatic composition.

[0011] 4. Main control and communication module: Used to control and coordinate communication between various modules. The STM32 microcontroller coordinates the modules, transmits EEG data via Bluetooth, and connects to mobile devices via WIFI.

[0012] 5. Power module, used to provide power to the entire device. It uses a lithium battery and supports long-term battery life.

[0013] The advantages of this invention are: it innovatively integrates EEG emotion perception and automatic music composition technology, breaking through the limitations of traditional preset music libraries; the wearable design improves the portability of the device and is suitable for everyday scenarios; the multi-algorithm fusion emotion recognition and music generation mechanism ensures the accuracy of emotion classification and the quality of music generation; and the real-time linkage mechanism realizes seamless connection between emotion changes and music adjustment, providing a brand-new solution for the fields of emotion regulation and music therapy.

[0014] Beneficial effects

[0015] The present invention has the following beneficial effects:

[0016] 1) Combining brainwave emotion perception with real-time automatic music composition to achieve personalized music generation.

[0017] 2) Highly practical, the wearable design is suitable for daily use and applicable to scenarios such as mood regulation and music therapy.

[0018] 3) Emotion recognition and music generation have low latency, meeting real-time requirements.

[0019] 4) More emotion categories and music styles can be expanded by updating the model and dataset. Attached Figure Description

[0020] Figure 1 It is a block diagram of the overall structure of the equipment.

[0021] Figure 2 This is the circuit diagram of the EEG acquisition module.

[0022] Figure 3 This is a flowchart of emotion recognition and music generation. Detailed Implementation

[0023] The following are specific embodiments of this patent.

[0024] First, assemble the wearable EEG acquisition module. Then, install the electrodes, carefully selecting 8-channel Ag / AgCl dry electrodes. Precisely place two electrodes at the FP1 and FP2 positions on the forehead, distribute four electrodes evenly in the mastoid region behind the ears, and fix the remaining two electrodes as reference electrodes on the earlobes. Ensure the electrode diameter is 3mm and the spacing between electrodes is strictly controlled at 5mm.

[0025] The frame is further integrated, and the installed electrodes are securely fixed to the headset frame using elastic bandages, ensuring that the wearing pressure does not exceed 0.5N, thereby ensuring wearing comfort and signal acquisition stability. During the fixing process, the electrode position needs to be checked repeatedly to avoid electrode displacement due to improper fixing, which could affect signal acquisition.

[0026] Further connecting the circuit, in the signal preprocessing circuit section, the 22kΩ resistor R52 and the 72nF capacitor C72 are correctly connected according to the circuit principle to form a pre-amplifier RC low-pass filter. The cutoff frequency of this filter can be determined using the formula... The calculated frequency is approximately 100Hz, and its function is to effectively filter out high-frequency noise.

[0027] Further configure the chip by performing detailed configuration of the analog front-end chip ADS1299, setting the gain to 24x and the sampling rate to 500Hz. Connect it to the main control chip via the SPI protocol, and simultaneously set register address 0x01 to 0x06 to enable the 8-channel acquisition function. During configuration, the error correction function of the programming tool should be used to carefully check whether all parameter settings are correct to avoid chip malfunction due to configuration errors.

[0028] To further power the device, a 3.7V lithium battery is used. First, the battery is connected to a TPS61021 boost converter chip to convert the voltage to 5V, specifically powering the ADS1299 chip. Then, an AMS1117-3.3V chip further converts the voltage to 3.3V to provide a stable operating voltage for other modules. After connecting the power supply, a multimeter should be used to check if the output voltage is stable within the specified range, and to ensure that the standby current does not exceed 10mA.

[0029] Next, configure the main control and communication modules. First, initialize the chip, selecting the STM32F103C8T6 as the main control chip and configuring it. Set the main frequency to 72MHz, enable the SPI1 interface, and connect it to the ADS1299, setting the clock polarity CPOL=0 and phase CPHA=1. Then, enable the USART2 interface to connect the ESP8266 WIFI module, setting the baud rate to 115200; and enable the USART3 interface to connect the HC-05 Bluetooth module, setting the baud rate to 9600. Set Timer 3 to a 10ms interrupt for data synchronization. The initialization process must strictly follow the chip's datasheet, ensuring accurate settings for all interfaces and timers by writing specific initialization code.

[0030] Further configure the Bluetooth module, setting it to slave mode and specifying each data packet as 20 bytes, including the channel number, timestamp, and 16-bit AD value. To prevent downtime during data transmission, set the packet interval to 100ms. During configuration, a Bluetooth configuration tool should be used to precisely set the module's parameters, and the data transmission stability of the Bluetooth module should be tested through simulated data transmission.

[0031] Further configure the Wi-Fi module by using AT commands to set it as a TCP client for connecting to the cloud server. Set the heartbeat interval to 30 seconds to ensure a stable connection with the server. During the setup process, carefully write the AT commands, send them to the Wi-Fi module via a serial communication tool, and monitor the module's response in real time to ensure successful configuration.

[0032] Next, install the music playback module. First, connect the chips using the BY8301 digital audio chip, communicating with the main control chip via USART1. This chip supports MP3 / WAV formats and has a sampling rate of 44.1kHz. During the connection process, strictly follow the chip's pin definitions for wiring to ensure correct connections.

[0033] Further instruction testing was conducted on the BY8301 chip. The 0xAA instruction was sent to control music playback / pause, and the 0x01-0x04 instructions were sent to adjust the volume (levels 1-4). Test code was written to send different instructions to the chip, and the music playback status and volume adjustment effects were observed to ensure the chip was functioning correctly.

[0034] Next, mount the storage module. First, attach a 4GB Micro SD card as the storage module and connect it to the main control chip. Partition the card to store music clips corresponding to four emotions (happy, sad, angry, and calm), with 100 clips in each category, each clip lasting 30 seconds. Name the files according to the format "emotion tag_serial number.mp3", for example, "happy_001.mp3". After mounting the storage module, use file system operation functions to read and write files on the card to test and ensure the storage module is working properly.

[0035] The integrated structure features an ABS headset frame that weighs only 45g, ensuring lightweight comfort. An adjustable headband with a range of 35-55cm is included to accommodate different head sizes.

[0036] Further, the electrode wires are cleverly integrated into the frame to prevent them from being exposed and tangled or damaged. During the installation process, the electrode wires must be properly secured and protected to ensure they are not damaged by shaking or friction within the frame.

[0037] The interface has been further designed with a magnetic attachment for easy disassembly and charging. When installing the magnetic interface, ensure the correct polarity and appropriate magnetic force to guarantee a stable connection while facilitating user disassembly.

[0038] Further EEG signal preprocessing and emotion recognition then commence. First, data acquisition and import are performed. After the device is powered on, the STM32 main control chip completes the initialization of each module, and the ADS1299 begins acquiring EEG signals. Data is acquired every 10 seconds, and the raw EEG signals are transmitted to the data processing module. During data transmission, the integrity and accuracy of the data must be ensured, and the data is verified using methods such as checksums.

[0039] Further bandpass filtering was performed using a 5th-order Butterworth bandpass filter to process the raw EEG signal, with a passband set to 1-100Hz. By writing filter algorithm code, the signal was filtered to effectively remove DC drift and power frequency interference, thus improving signal quality.

[0040] To further remove artifacts, Independent Component Analysis (ICA) is used to remove artifacts from the bandpass-filtered signal. The signal is input into the ICA algorithm model, and through iterative calculations, artifact components such as electrooculography (EOG) and electromyography (EMG) are separated, while the valid EEG components are retained. During algorithm implementation, the algorithm parameters need to be optimized according to the signal characteristics to improve the artifact removal effect.

[0041] Further feature extraction was performed, and the Welch method was used to calculate the power spectral density of the processed EEG signal. The window length was set to 2 seconds, the overlap rate to 50%, and the FFT point count to 1024. Using these parameters, the relative powers of the δ (1-4Hz), θ (4-8Hz), α (8-13Hz), β (13-30Hz), and γ (30-100Hz) waves were extracted as feature values. During the calculation process, specialized signal processing library functions were used to ensure the accuracy of the results.

[0042] Further training and classification were conducted on the model using 75% of the DEAP dataset (containing data from 30 subjects) and a self-collected dataset (13 subjects, 100 samples per emotion), totaling 10,000 samples for training the four-class classification model. The model integrated algorithms such as KNN (K=5), SVM (RBF kernel, C=10, γ=0.1), decision tree (CART algorithm), and random forest (100 trees). A voting method was used to comprehensively judge the outputs of multiple models to obtain the final emotion classification result. During model training, the hyperparameters of the model needed to be optimized, and methods such as cross-validation were used to improve the model's accuracy and generalization ability. The trained model was deployed to a server to classify emotions from real-time acquired EEG signal features, outputting emotion labels (0 for happiness, 1 for sadness, 2 for anger, and 3 for calm), with the entire process taking ≤500ms.

[0043] Next, the automatic composition module was trained. First, raw data was created by collecting 500 pieces of music already labeled with emotions, including 200 classical pieces, 150 new age pieces, and 150 natural sound pieces. These pieces were then analyzed using professional music analysis software, converting them into sequences of pitch (0-127), duration (0.25-4 beats), and dynamics (0-127). During the analysis process, the accuracy of the results was ensured, and the analyzed sequences underwent quality checks.

[0044] Further supplementary data was generated using a genetic algorithm. The initial population consisted of 200 random sequences. A fitness function was constructed, considering the cosine similarity between musical features and emotional labels (i.e., emotional correlation), the probability of the musical sequence appearing in the original dataset (e.g., the occurrence of consecutive pitch sequences like 2-grams and 3-grams), and dissonances that violated musical rules (e.g., excessive interval jumps). The crossover probability was set to 0.8 and the mutation probability to 0.05. After 50 generations of iteration, 10,000 supplementary data sets were generated. During the implementation of the genetic algorithm, each step was meticulously recorded and debugged to ensure the quality of the generated data.

[0045] Next, train the LSTM network. First, build the network by constructing an LSTM network. The network structure includes an input layer (containing 32-dimensional features), a hidden layer (containing 256 neurons using the ReLU activation function), and an output layer (outputting a 1-dimensional emotion matching score). Use a deep learning framework (such as TensorFlow or PyTorch) to build the network, defining the structure and parameters of each layer according to the framework's specifications.

[0046] Further training parameters were configured using the Adam optimizer with a learning rate of 0.001 and a batch size of 32, for 1000 iterations. The loss function was defined as the difference between the predicted and true values ​​(e.g., mean squared error). During training, the changes in the loss function were monitored in real time, and the training parameters were adjusted according to the trend to improve the model's training performance. Ultimately, the validation set loss was optimized to approximately 0.12 to ensure the model's accuracy and stability.

[0047] Then, real-time linkage functionality is implemented. First, data acquisition and transmission are performed. After the device is powered on and initialized, the EEG acquisition module generates a feature vector every 10 seconds at a set frequency. This feature vector is sent to the server via Bluetooth. During transmission, the data needs to be packaged and encrypted to ensure the security and stability of data transmission.

[0048] To further identify emotions, the server receives the feature vector and immediately calls a pre-trained four-class classification model to process it and output an emotion label. For example, when the model determines that the user's current emotion is "anger," it returns the corresponding emotion label to the device.

[0049] To further generate music, the server, upon receiving the emotion tag, invokes the automatic music composition module. First, the LSTM network generates an initial musical framework based on the emotion tag, determining basic elements such as rhythm and key. Then, a genetic algorithm optimizes the pitch sequence based on the framework generated by the LSTM. Taking "anger" as an example, the generated music might correspond to a relatively fast tempo of 120 BPM, a minor key, and strong dynamics, ultimately producing a 30-second music clip. The entire music generation process must be completed within ≤2 seconds.

[0050] The system further plays music and records emotional changes, transmitting the generated music data back to the device via a Wi-Fi module. The device's BY8301 digital audio chip decodes and plays the music data. Simultaneously, the main control module begins recording an emotional change curve, updating the curve data every 30 seconds to record how the user's emotions change over time.

[0051] Further dynamic adjustments are made. If the user's emotion label changes, for example from "anger" to "calm," the device sends the new emotion label to the server. The server then calls the automatic music composition module again to adjust the music parameters according to the new emotion label. For example, the tempo BPM is reduced to 80, the key is changed to a major key, and a new music segment suitable for the new emotion is generated and sent back to the device for playback, achieving real-time dynamic adaptation between music and emotion.

[0052] Next, algorithm performance testing and adjustments are performed. First, an emotion recognition accuracy test is conducted using an independent test dataset containing EEG signal samples of various emotional states. The algorithm's accuracy in recognizing emotions such as happiness, sadness, anger, and calm is calculated, ensuring an average accuracy of over 82.6%. If the accuracy does not meet the requirements, the algorithm's model structure, parameter settings, or training data need to be optimized, such as increasing the amount of training data or adjusting model hyperparameters.

[0053] Further testing and adjustments to music adaptation were made by inviting 30 listeners (15 professional musicians and 15 non-professionals) to rate the emotional relevance of the generated music. Listeners were shown music generated under different emotional tags and asked to rate the degree of match between the music and their emotions based on their own feelings, with a maximum score of 5. The average score was calculated, ensuring it reached at least 3.8 points, significantly higher than the score for randomly played music (2.1 points). If the music adaptation was unsatisfactory, the algorithm of the automatic music composition module needed to be optimized, adjusting the rules and parameters of music generation to make the generated music more closely match the user's emotions.

[0054] Next, conduct user experience testing. First, conduct a comfort test, inviting 10 participants to wear the device continuously for 2 hours. During this time, participants provide real-time feedback on their wearing experience. After the test, participants rate the device's comfort level, with a maximum score of 5. Ensure the average comfort score reaches 4.2 or higher. If the score is low, improvements need to be made to the device's structural design, material selection, or wearing method to enhance user comfort.

[0055] Further testing and optimization of operation response latency were conducted. During device operation, user emotional changes were simulated, and the total latency from the occurrence of the emotional change to the completion and playback of music adjustment was measured. The total latency was ensured to be ≤3 seconds to meet real-time requirements. If the response latency was too long, the device's hardware performance, data transmission speed, or software algorithm execution efficiency needed to be optimized to shorten the response time and improve the user experience.

[0056] This invention integrates EEG technology with automatic music composition algorithms to provide users with a brand-new interactive experience between emotions and music, which has significant practical value and market prospects.

Claims

1. A wearable emotion-sensing automatic music composition device based on electroencephalogram (EEG) signals, characterized in that, It includes the following six modules. (1) Wearable EEG acquisition module: The ADS1299 analog front-end chip is used to acquire the user's EEG signal through 8-channel electrodes. The front-end RC low-pass filter removes high-frequency noise, and the sampling rate is 500Hz. (2) Emotion Recognition Module: A four-classification model (integrating KNN, SVM, decision tree, and random forest) is built based on the PyTorch framework. The power spectral density features of δ (1-4Hz), θ (4-8Hz), α (8-13Hz), β (13-30Hz), and γ (30-100Hz) waves in the EEG signal are extracted using the Welch method to classify four emotions: happiness, sadness, anger, and calmness, with an accuracy of no less than 82%. (3) Automatic music composition module: including LSTM network submodule and genetic algorithm optimization submodule. The LSTM network takes the music data training set (including original music and music generated by genetic algorithm) as input and takes the emotion matching degree as output target. The fitness function of the genetic algorithm is the weighted sum of emotion correlation degree and music rule score. (4) Music playback module: It adopts the BY8301 digital audio chip, supports MP3 and WAV formats, and integrates a power amplifier to directly drive the headphones. (5) Main control module: Based on STM32F103C8T6 microcontroller, it communicates with the EEG acquisition module through SPI protocol and realizes data transmission through Bluetooth and WIFI: controlling the coordinated work of each module. (6) Power module: It adopts a rechargeable lithium battery with a power supply voltage of 3.3V-5V, and supports continuous operation for more than 8 hours.

2. The device according to claim 1, characterized in that, The wearable EEG acquisition module is integrated into the earpiece structure, with electrodes placed behind the ear, on the mastoid process, and on the forehead. It weighs no more than 50g and is waterproof and dustproof (IP54 rating).

3. The device according to claim 1, characterized in that, The four-class classification model of the emotion recognition module is jointly trained using the DEAP dataset and the self-collected EEG dataset. The self-collected dataset contains EEG data of 13 subjects under different emotion induction conditions, with no less than 100 samples for each emotion.

4. The device according to claim 1, characterized in that, The method for constructing the music data training set of the automatic music composition module is as follows: The original music dataset contains classical, new age, and other genres of music, categorized by emotion labels. A genetic algorithm is used to generate supplementary music, with a fitness function M = S_emotional association + S2-gram + S3-gram - S_dissonance, where S_emotional association represents the degree of matching between the music features and the target emotion, S2-gram and S3-gram are the frequency of occurrences of pitch sequences in the original dataset, and S_dissonance is the number of times music rules are violated.

5. The device according to claim 1, characterized in that, The steps for constructing the music data training set for the automatic music composition module are as follows: (1) Wearable device initialization: The EEG acquisition module starts to acquire the user's EEG signal. The signal is preprocessed by RC low-pass filter and Butterworth band-pass filter to remove 50Hz power frequency interference and electromyography noise. (2) The emotion recognition module uses the Welch method to calculate the power spectral density of each frequency band of the EEG signal and inputs it into the four-classification model to obtain the real-time emotion category (happy / sad / angry / calm); (3) The automatic music composition module calls the LSTM network with corresponding parameters according to the emotion category, with the goal of maximizing the emotion matching degree, and combines the genetic algorithm to optimize the pitch sequence, rhythm and chord progression to generate a music segment with a duration of 1-3 minutes. (4) The music playback module plays the generated music in real time, while the main control module uploads the emotion data and the generated music to the mobile terminal for storage via WIFI; (5) Continuously collect EEG signals, update the emotion recognition results every 30 seconds, dynamically adjust the music generation parameters, and realize real-time adaptation of emotions and music.

6. The device according to claim 1, characterized in that, The method for constructing the music data training set for the automatic composition module is as follows: The original music dataset contains music of various styles such as classical and new age, and is classified according to emotion labels. Supplementary music is generated using a genetic algorithm, with a fitness function M = S_emotion association + S2-gram + S3-gram - S_dissonance, where S_emotion association is the degree of matching between the music features and the target emotion, S2-gram and S3-gram are the frequency of occurrence of pitch sequences in the original dataset, and S_dissonance is the number of times music rules are violated. The original music dataset contains classical, new age, and other genres of music, categorized by emotion labels. A genetic algorithm is used to generate supplementary music, with a fitness function M = S_emotional association + S2-gram + S3-gram - S_dissonance, where S_emotional association represents the degree of matching between the music features and the target emotion, S2-gram and S3-gram are the frequency of occurrences of pitch sequences in the original dataset, and S_dissonance is the number of times music rules are violated.

7. The device according to claim 1, characterized in that, The method for constructing the music data training set of the automatic music composition module is as follows: The original music dataset contains classical, new age, and other genres of music, categorized by emotion labels. A genetic algorithm is used to generate supplementary music, with a fitness function M = S_emotional association + S2-gram + S3-gram - S_dissonance, where S_emotional association represents the degree of matching between the music features and the target emotion, S2-gram and S3-gram are the frequency of occurrences of pitch sequences in the original dataset, and S_dissonance is the number of times music rules are violated.

Images