A method, device, equipment and medium for generating brainwave music

Abstract

The present application discloses a method, apparatus, device and medium for generating EEG music, and relates to the field of artificial intelligence technology. The method comprises: obtaining an EEG emotion dataset, an emotion music dataset and an EEG music pairing dataset; obtaining a pre-trained EEG-to-emotion mapping module according to the EEG emotion dataset; obtaining a pre-trained emotion-to-music mapping module according to the emotion music dataset; fine-tuning the pre-trained model using a Mamba network according to the EEG music pairing dataset to obtain an EEG music generation model; utilizing the EEG emotion dataset and the emotion music dataset to compensate for the problem of insufficient EEG music pairing data, thereby enhancing the ability to control emotions while retaining the high-quality effect of music generated by the diffusion model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for generating brainwave music. Background Art

[0002] Advances in computer applications and brain-computer interface (BCI) systems have led to the gradual application of brainwave music generation technology in fields such as music creation and music therapy. Electroencephalogram (EEG) signals are an objective record of how people experience music, reflecting the impact of music on the brain. By studying the correlation between EEG signals and musical content, EEG music generation methods based on neural network architectures can learn complex, generalized nonlinear mapping relationships from large amounts of data. Large-scale, high-quality datasets can enable deep learning models to generalize better, with greater robustness and accuracy. However, the insufficiency and difficulty in acquiring EEG-to-music paired datasets have had a fatal impact on deep learning-based EEG-to-music generation.

[0003] Therefore, how to construct a method to generate EEG to music, ensure the quality of the generated music, and use music as an art form to express the brain's physiological state information carried by EEG signals has always been an important topic in neuroscience. Summary of the Invention

[0004] The purpose of this application is to provide a method, device, equipment and medium for generating brainwave music, which can enhance the emotional control ability of brainwave music generation.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a method for generating brainwave music, the method comprising:

[0007] Obtain an EEG emotion dataset, an emotional music dataset, and an EEG music pairing dataset; the EEG emotion dataset and the emotional music dataset are open source datasets, and the EEG music pairing dataset is constructed by collecting EEG signals under emotional music stimulation;

[0008] Training a fusion network based on the EEG emotion dataset to obtain a pre-trained EEG to emotion mapping module; the fusion network is a fusion network including a convolutional neural network and a long short-term memory network;

[0009] An improved diffusion model is trained based on the emotional music dataset to obtain a pre-trained emotion-to-music mapping module; the improved diffusion model is used to generate music under the conditional control using the musical emotion as a conditional control;

[0010] According to the EEG music pairing dataset, the pre-trained model is fine-tuned to obtain an EEG music generation model; the pre-trained model includes the EEG to emotion mapping module, the Mamba network and the emotion to music mapping module; the Mamba network is used to generate music emotion information based on the EEG emotion information generated in the EEG to emotion mapping module, and input the music emotion information into the emotion to music mapping module; the EEG music generation model is used to generate music based on EEG signals.

[0011] Optionally, the EEG emotion dataset includes a DEAP dataset, and the emotional music dataset includes an EMOPIA dataset and a Lakh dataset.

[0012] Optionally, training a fusion network based on the EEG emotion dataset to obtain a pre-trained EEG to emotion mapping module specifically includes:

[0013] Slicing the EEG signal of each sample in the EEG emotion dataset using a time window of a set length, extracting features from the EEG signal of each slice using a domain transformation method, and obtaining a two-dimensional feature time frame of the EEG signal;

[0014] The fusion network is trained with the two-dimensional feature time image frames of each EEG signal as input and the corresponding EEG emotion as output to obtain a pre-trained EEG-to-emotion mapping module;

[0015] The fusion network also includes a first classifier. The convolutional neural network in the fusion network is used to extract a first feature sequence including spatial information of the EEG signal from the two-dimensional feature time frame. The long short-term memory network is used to extract a second feature sequence including spatial information and time information of the EEG signal from the first feature sequence. The first classifier is used to perform EEG emotion recognition based on the second feature sequence.

[0016] Optionally, training the improved diffusion model based on the emotional music dataset to obtain a pre-trained emotion-to-music mapping module specifically includes:

[0017] The symbolic music feature processing module in MusicVAE is used to perform data processing and feature extraction on the music signal of each sample in the emotional music dataset to obtain a one-dimensional pitch time series of a single-track melody;

[0018] The one-dimensional pitch time series of the single-track melody is sliced ​​using a time window of set duration to obtain a one-dimensional pitch time frame;

[0019] A transformer neural network is used to capture the global dependencies and contextual information of the pitch time series of a single-track melody, obtaining a feature sequence of musical emotion information.

[0020] The second classifier is used to perform emotion recognition on the music emotion information feature sequence;

[0021] The improved diffusion model is trained by taking the music emotion information feature sequence as the conditional control information of the improved diffusion model to obtain a pre-trained emotion-to-music mapping module.

[0022] Optionally, the pre-trained model is fine-tuned based on the EEG music pairing dataset to obtain an EEG music generation model, specifically including:

[0023] During fine-tuning of the pre-trained model according to the EEG music pairing dataset, the second feature sequence output by the long and short-term memory network in the EEG to emotion mapping module is input into the Mamba network, and the music emotion information feature sequence output by the Mamba network is input into the emotion to music mapping module.

[0024] Optionally, the improved diffusion model is to add a cross-attention module between the multi-head self-attention layer and the feedforward neural network in the transformer block of the original diffusion model, wherein the query vector in the cross-attention module inputs the melody sequence information output by the multi-head self-attention layer, and the key vector and value vector in the cross-attention module input the music emotion information feature sequence.

[0025] Optionally, based on the EEG music pairing dataset, the loss function in the process of fine-tuning the pre-trained model is expressed as:

[0026] L t =λ1L rf +λ2L rvb ;

[0027] Among them, L t is the overall loss function value, λ1 and λ2 are weight parameters, L rf For emotional loss, L rvb Loss of music quality.

[0028] In a second aspect, the present application provides a device for generating brainwave music, the device comprising:

[0029] The dataset acquisition module is used to obtain EEG emotion datasets, emotion music datasets, and EEG music pairing datasets;

[0030] A first pre-training module is used to train a fusion network based on the EEG emotion dataset to obtain a pre-trained EEG to emotion mapping module; the fusion network is a fusion network including a convolutional neural network and a long short-term memory network;

[0031] A second pre-training module is used to train an improved diffusion model based on the emotional music dataset to obtain a pre-trained emotion-to-music mapping module; the improved diffusion model is used to use music emotions as conditional control to generate music under the conditional control;

[0032] A model fine-tuning module is used to fine-tune the pre-trained model according to the EEG music pairing dataset to obtain an EEG music generation model; the pre-trained model includes the EEG to emotion mapping module, the Mamba network and the emotion to music mapping module; the Mamba network is used to generate music emotion information based on the EEG emotion information generated in the EEG to emotion mapping module, and input the music emotion information into the emotion to music mapping module; the EEG music generation model is used to generate music based on EEG signals.

[0033] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any one of the above-described methods for generating brainwave music.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the above-mentioned methods for generating brainwave music.

[0035] According to the specific embodiments provided in this application, this application discloses the following technical effects: This application provides a method, device, equipment and medium for generating EEG music, obtains an EEG emotion data set, an emotion music data set and an EEG music pairing data set, obtains a pre-trained EEG to emotion mapping module based on the EEG emotion data set, obtains a pre-trained emotion to music mapping module based on the emotion music data set, and uses the Mamba network to fine-tune the pre-trained model based on the EEG music pairing data set to obtain an EEG music generation model. The use of the EEG emotion data set and the emotion music data set can make up for the problem of insufficient EEG music pairing data, while retaining the high-quality effect of the diffusion model to generate music, and enhance the emotional control ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 A flowchart of a method for generating brainwave music provided in one embodiment of the present application;

[0038] Figure 2 A flowchart of a method for generating EEG emotional music based on pre-training model fine-tuning technology provided in one embodiment of the present application;

[0039] Figure 3 A schematic diagram of a method for generating EEG emotional music based on a pre-training model fine-tuning technique according to an embodiment of the present application;

[0040] Figure 4 A flowchart of the process of obtaining an EEG music pairing dataset provided in one embodiment of the present application;

[0041] Figure 5 A schematic diagram of the functional modules of a device for generating brainwave music according to an embodiment of the present application;

[0042] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0044] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0045] This application provides a method for generating brainwave music. Figure 1 As shown, the brainwave music generation method includes:

[0046] Step 101: Obtain an EEG emotion dataset, an emotional music dataset, and an EEG music pairing dataset; the EEG emotion dataset and the emotional music dataset are open source datasets, and the EEG music pairing dataset is constructed by collecting EEG signals under emotional music stimulation.

[0047] Step 102: training a fusion network based on the EEG emotion dataset to obtain a pre-trained EEG-to-emotion mapping module; the fusion network is a fusion network including a convolutional neural network and a long short-term memory network.

[0048] Step 103: training an improved diffusion model based on the emotional music dataset to obtain a pre-trained emotion-to-music mapping module; the improved diffusion model is used to use music emotions as conditional control to generate music under the conditional control.

[0049] Step 104: Fine-tune the pre-trained model according to the EEG music pairing dataset to obtain an EEG music generation model; the pre-trained model includes the EEG to emotion mapping module, the Mamba network and the emotion to music mapping module; the Mamba network is used to generate music emotion information based on the EEG emotion information generated in the EEG to emotion mapping module, and input the music emotion information into the emotion to music mapping module; the EEG music generation model is used to generate music based on EEG signals.

[0050] In an exemplary embodiment, the EEG emotion dataset includes the DEAP dataset, the emotional music dataset includes the EMOPIA dataset and the Lakh dataset, and each sample in the EEG music pairing dataset includes an EEG signal and a music signal that stimulates the brain, wherein the EEG signal is an EEG signal generated under the stimulation of the music signal.

[0051] This application uses a time window of set length, and applies domain transformation methods and symbolic music processing technology to perform feature extraction processing on EEG and music data, obtaining a two-dimensional feature time graph frame of EEG and a pitch time series of a single-track melody; the domain transformation method includes: time domain transformation, frequency domain transformation, and time-frequency domain transformation, and domain transformation is used to perform feature mapping processing on the one-dimensional feature sequence to obtain a two-dimensional feature time graph frame; the symbolic music processing technology is taken from the music feature processing module in MusicVAE. MusicVAE is a model based on variational autoencoders, specifically used for music generation and representation learning.

[0052] The EEG emotion dataset and the emotional music dataset are both large datasets.

[0053] In an exemplary embodiment, step 102 specifically includes:

[0054] The EEG signal in each sample in the EEG emotion dataset is sliced ​​using a time window of a set length, and the EEG signal of each slice is feature extracted using a domain transformation method to obtain a two-dimensional feature time frame of the EEG signal. More specifically, the EEG signal in each sample in the EEG emotion dataset is first filtered and artifacts are removed to obtain denoised EEG data; the denoised EEG data is resampled to obtain EEG signal resampled data, and the EEG signal resampled data is sliced ​​using a time window of a set length, and the EEG signal of each slice is feature extracted using a domain transformation method (feature mapping is performed on the one-dimensional feature sequence) to obtain a two-dimensional feature time frame of the EEG signal.

[0055] The fusion network is trained using the two-dimensional feature time graph frames of each EEG signal as input and the corresponding EEG emotion as output to obtain a pre-trained EEG-to-emotion mapping module. The pre-trained EEG-to-emotion mapping module can convert the input EEG signal into a time series of emotional features.

[0056] The fusion network also includes a first classifier. The convolutional neural network in the fusion network is used to extract a first feature sequence including spatial information of the EEG signal from the two-dimensional feature time frame. The long short-term memory network is used to extract a second feature sequence including spatial and temporal information of the EEG signal from the first feature sequence. The first classifier is used to perform EEG emotion recognition based on the second feature sequence. The spatiotemporal feature sequence (second feature sequence) of the EEG emotion information output from the middle part of the pre-training model is intercepted and used as an EEG emotion information extraction module for subsequent fine-tuning of the pre-training model.

[0057] Convolutional neural networks extract local features from the input four-dimensional EEG data through convolution operations in convolutional layers. The convolution operation uses multiple convolution kernels to scan the input data, generating multiple feature maps. Each convolution kernel is a 3×3 or 5×5 sliding matrix that calculates a weighted sum of a local region and outputs it to the feature map. Through multiple convolutional layers, more complex high-level features are gradually learned. An activation function is then used to introduce nonlinearity and alleviate the vanishing gradient problem. Finally, the features extracted by the convolutional layer are downsampled through the pooling layer to reduce the spatial size of the feature map, thereby reducing the computational effort and preventing overfitting.

[0058] The LSTM network is used to capture the contextual information of the sequence, obtaining an output feature sequence that extracts the spatial and temporal information of the EEG signal. The LSTM network is specifically a bidirectional LSTM subnetwork.

[0059] In an exemplary embodiment, an improved conditional control absorbing state diffusion model is used on a large emotional music dataset to obtain a pre-trained emotion-to-music mapping module; the improved conditional control absorbing state diffusion model uses the emotional information of the music as a conditional control and completes the conditional control generation of emotional music through training. Step 103 specifically includes:

[0060] The symbolic music feature processing module in MusicVAE is used to perform data processing and feature extraction on the music signal in each sample in the emotional music dataset to obtain a one-dimensional pitch time series of a single-track melody.

[0061] The one-dimensional pitch time series of the single-track melody is sliced ​​using a time window of set duration to obtain a one-dimensional pitch time frame.

[0062] The transformer neural network is used to capture the global dependencies and contextual information of the pitch time series of a single-track melody, and obtain a feature sequence of musical emotion information.

[0063] The second classifier is used to perform emotion recognition on the music emotion information feature sequence.

[0064] The improved diffusion model is trained using the music emotion information feature sequence as the conditional control information of the improved diffusion model to obtain a pre-trained emotion-to-music mapping module. This application intercepts the music emotion information feature sequence output in the middle part of the pre-trained model as the conditional control information of the improved absorbing state diffusion model.

[0065] The first classifier and the second classifier are both binary classifiers.

[0066] The purpose of using the symbolic music feature processing module in MusicVAE is to process the symbolic music features and extract a 4 / 4 (quarter note as one beat, each measure has 4 beats) monophonic melody with a single sample length of 8 bars from the data.

[0067] In an exemplary embodiment, the improved diffusion model adds a cross-attention module between the multi-head self-attention layer and the feedforward neural network in the transformer block of the original diffusion model, wherein the query vector in the cross-attention module inputs the melody sequence information output by the multi-head self-attention layer, and the key vector and value vector in the cross-attention module input the music emotion information feature sequence.

[0068] The diffusion model is an absorption-state diffusion model. During the pre-training phase of the diffusion model for emotional conditioning, an open-source music dataset was used. The input for forward diffusion was music, and the output was noisy music. The input for reverse diffusion was the noisy music and the corresponding emotional information extracted from the current music, and the output was the restored music.

[0069] In one exemplary embodiment, a pre-trained model is fine-tuned using a Mamba network and a specialized loss function on a small, self-developed dataset, namely, an EEG-to-music pairing dataset, to complete the mapping of EEG to emotion and then to music. Step 104 specifically includes: during the fine-tuning of the pre-trained model based on the EEG-music pairing dataset, the second feature sequence output by the long-short-term memory network in the EEG-to-emotion mapping module is input into the Mamba network, and the music emotion information feature sequence output by the Mamba network is input into the emotion-to-music mapping module.

[0070] The Mamba network is a state-of-the-art selective structural state-space model (SSM). It has linear scalability and adaptability across multiple modalities. The Mamba network selectively processes the input EEG time series and converts it into a single-track melody emotional information sequence as an output. Specifically, the input data is first layer-normalized and then feature extracted using a multi-layer perceptron (MLP). Next, these features are further processed through deep convolution to enhance the connectivity of local features, generating intermediate results. After being processed by the SiLU activation function, the features are refined through the SSM module and residual connections are added.

[0071] In one exemplary embodiment, the loss function used in fine-tuning a pre-trained model based on an EEG-music pairing dataset is a weighted sum of a music quality loss sub-function and an emotion bias sub-function. This loss function evaluates the output of the pre-trained model and fine-tunes it using gradient descent to reduce information bias between different datasets, ultimately achieving a mapping from EEG to emotion and then to music.

[0072] This application expresses the emotional information encoded by the emotion mapping module from the EEG signal as The output of the melody emotion information extractor is taken as the reference truth value, denoted as E i , where i represents the time step. To measure the difference between the two, the reference loss is defined as:

[0073] In order to fine-tune the generated distribution while pre-training the guidance emotion information mapping module, Re-estimated Evidence Lower Bound (ELBO), the formula is as follows:

[0074]

[0075] The inner sum in the above formula forms the given x in x0 t The joint (log) probability of all random variables in , thus obtaining x0 and x t Between E q(xt|x0) Expected cross entropy loss.

[0076] Among them, x0 represents the original data, x t represents the data obtained by adding noise to the original data x0 at time step t, q(x t |x0) means that under the condition of given original data x0, the noise data x is obtained. t The probability distribution when It means that under the condition of known original data x0, after the noise adding process of time step t, x t The average value of p θ ([x0]i |x t ) represents the noise data x at a given time step t t Under the condition of, the probability distribution of the i-th element of the original data x0, E q(x0) represents the expectation of the probability distribution q(x0), T represents the maximum number of time steps allowed in the forward diffusion process, [x0] i Represents the i-th element of the original data x0.

[0077] The loss function is expressed as: L t =λ1L rf +λ2L rvb .

[0078] Among them, L t is the overall loss function value, λ1 and λ2 are weight parameters, λ1=0.2, λ2=1, L rf For emotional loss, L rvb Loss of music quality.

[0079] In an exemplary embodiment, Figure 2 and Figure 3 As shown, a method for generating brainwave music includes:

[0080] First, we obtain open-source DEAP datasets, EMOPIA datasets, Lakh datasets, etc.; obtain the subjects' EEG signal data and music files that stimulate the brain through experiments, and evaluate the effectiveness of all the acquired data.

[0081] Experts were organized to select music clips that could induce the corresponding emotional states of the subjects, analyze the application scenarios and possible situations, and design a relatively complete paradigm that could accurately reflect brain activity under emotional music stimulation.

[0082] While isolating the subjects from external interference, the discharge signals of the cerebral cortex were recorded and collected when they were stimulated by music with different emotions.

[0083] We analyzed participants' performance under different emotional music stimulation conditions, collected emotional ratings for each stimulus, and evaluated the quality of the collected EEG signals. We removed raw data with poor performance and excessive noise to construct a small dataset of raw EEG and music pairings.

[0084] The collected raw EEG signal data is preprocessed to extract target domain features, and the paired emotional music files are preprocessed. Specifically, the following steps are performed: using the filtering module to perform bandpass filtering on the retained EEG signal, retaining the effective frequency band and removing irrelevant information; using the artifact removal module to process noise interference signals in the EEG signal, such as electro-oculography, electrocardiography, and electromyography; and using the resampling module to downsample the data. The preprocessing process includes:

[0085] 1) Use a bandpass filter to isolate the effective EEG signals within four frequency bands: theta (4-7 Hz), alpha (8-13 Hz), beta (14-30 Hz), and gamma (31-50 Hz). Filter out high-frequency and low-frequency noise interference.

[0086] 2) Remove physiological signal interference such as electrooculographic artifacts, electromyographic artifacts and 50hz power frequency interference.

[0087] 3) Downsample the retained EEG signals to a sampling frequency of 128 Hz, improving computational efficiency while preserving brain state information.

[0088] The one-dimensional EEG data is added with a time window, and the EEG features in the target domain are extracted to form a feature set, which is then converted into a two-dimensional feature time frame based on the electrode distribution.

[0089] 4) In order to align the processed EEG feature map frames and pitch sequences in time, each paired EEG signal and stimulation music data was cut into 16 seconds, and a sliding time window of 0.5 seconds (the set time for slicing, that is, the time length of the slicing window) was used to capture task-related information in long segments (feature extraction of each electrode channel signal), and there was no overlap between adjacent windows.

[0090] Calculate the time domain characteristics of the EEG signal, including statistical characteristics and nonlinear characteristics; separate the five relevant frequency bands recognized in the field and calculate their respective power spectral density and differential entropy characteristics; use short-time Fourier transform to divide the signal into subsequences and extract their time-frequency domain characteristics.

[0091] By comparing the accuracy and significance indicators of various features in EEG emotion tasks, the one-dimensional differential entropy feature sequence extracted above is selected as the feature for subsequent tasks. The features extracted from each time window are mapped to a two-dimensional matrix that conforms to the distribution of EEG cap electrodes to form a two-dimensional feature time frame.

[0092] First, the symbolic music processing submodule (symbolic music feature processing module) of MusicVAE is used to process the music sequence that matches the retained EEG signal and extract the one-dimensional pitch time series of the corresponding single-track melody. Then, the mean, absolute value of the first-order difference, variance, and Hjorth parameter of the EEG signal data are calculated in the time domain. The power spectral density and differential entropy are calculated in the four frequency bands of θ, α, β, and γ of the EEG signal. In the time-frequency domain, wavelet transform is applied to extract features. The calculation formulas that need to be explained in the above features are shown in Table 1 below.

[0093] Table 1 Characteristic calculation formula

[0094]

[0095]

[0096] Where N is the number of samples, var is the variance, E is the average power, x' n is x n First derivative, x n is the nth sampling point, σ represents the standard deviation, d j,k represents the wavelet coefficient of the original signal at scale j and translation position k, h j,k-2n represents the low-pass filter coefficient, a j,k represents the scale factor, g j,k-2n Represents the coefficient of the high-pass filter, and x[n] represents the value of the original signal at discrete time n.

[0097] 5) Preliminary EEG emotion recognition task experiments and significance difference analysis were conducted using one-dimensional sequences with different features. The comparison showed that the single feature with the best effect in the EEG emotion recognition task was differential entropy.

[0098] The one-dimensional differential entropy features of the EEG signal are converted into two-dimensional feature time frames. Specifically, the process involves converting the one-dimensional differential entropy feature sequence into a two-dimensional feature time frame based on the electrode distribution of the EEG cap and normalizing the non-zero elements. The eigenvalues ​​of each time window are mapped to an 8×9 matrix according to their corresponding positions, forming a two-dimensional feature time frame.

[0099] On the open-source large-scale EEG emotion dataset collected above, for the differential entropy feature, its two-dimensional feature time graph frame is used as the input of the volume and long short-term memory fusion network. The network will extract and integrate its spatial information and time information to obtain a spatiotemporal feature sequence (second feature sequence) representing the EEG emotion information. This sequence will be used as the input of the first classifier to pre-train the EEG emotion recognition task. The spatiotemporal feature sequence of the EEG emotion information output in the middle part of the pre-training model is intercepted as the EEG emotion information extraction module for subsequent fine-tuning of the pre-training model. Specifically including:

[0100] 1) Superimpose the four frequency bands of the two-dimensional feature time frame obtained above that retains the EEG spatial topology, and take c adjacent sliding window slices as a training sample. Each sample can be represented as a four-dimensional structure X n ∈R c×d×h×w , n=1,2,...,N. Where N is the total number of samples, h and w are the height and width of the two-dimensional feature map respectively, d is the number of frequency bands, and c is the number of time frames that make up a sample, which is set to 4 in this application. n∈{HA,LA,HV,LV} represents the classification labels of these EEG sequences, where HA represents high arousal, LA represents low arousal, HV represents high valence, and LV represents low valence.

[0101] 2) Input the processed features into the convolutional neural network to obtain an output feature sequence that extracts the spatial information of the EEG signal.

[0102] 3) Use the first classifier to pre-train the output spatiotemporal feature sequence for EEG emotion recognition. Intermediate outputs of the pre-trained model are intercepted to extract emotion information vectors from consecutive small time slices of a long time series, ultimately obtaining emotion information.

[0103] On the large-scale emotional music dataset collected above, the one-dimensional pitch time series of a single-track melody, pre-processed by the MusicVAE submodule, is used as the input to the transformer neural network. This is used to capture the global dependencies and contextual information of the pitch time series of the single-track melody, thereby obtaining a sequence of musical emotional information features. This sequence is used as the input to the second classifier to pre-train the music emotion recognition task. The music emotion information feature sequence output by the middle part of the pre-trained model is intercepted and used as the conditional control information of the improved absorbing state diffusion model to obtain a pre-trained emotion-conditioned music diffusion generation module, which is subsequently used for model fine-tuning.

[0104] Using the improved conditionally controlled absorbing state diffusion model, we pre-trained an emotion-controlled music generation module on a large emotional music dataset. The steps include:

[0105] 1) The one-dimensional pitch time series of the processed single-track melody is input into the absorbing state diffusion model, and a cross-attention mechanism is used to weight the input at each time step. A cross-attention module is added between the multi-head self-attention layer and the feedforward neural network in the transformer block of the original diffusion model. The query vector q is derived from the hidden layer representation of the melody sequence output by the previous layer, while the key vector k and value vector v are derived from the melody emotion information output by the corresponding pre-trained module. Through the multi-head cross-attention mechanism, the attention weights between the query vector and the key vector are calculated and applied to the value vector. It can be expressed as:

[0106] MultiHead(M',F M ,F M )=Concat(head1,…,head h )W O ;

[0107]

[0108] Among them, W O , and is a learnable weight matrix, and M' is the melody sequence output by the middle hidden layer of the model. MultiHead(M',F M ,F M ) is a multi-head self-attention layer, Concat means connection, head i represents the i-th attention head, CrossAttention represents cross attention, F M is the music emotion information feature sequence output by the music emotion extractor, F M ∈R N×T×Z , R N×T×Z It is data of N×T×Z dimensions, where N is the number of data points in each time step, T is the time step, and Z is the number of sequences.

[0109] 2) For the key vector k and value vector v mentioned above, they are taken from the intermediate output of the pre-trained melody emotion extractor (emotion to music mapping module), namely F M ,The extractor is constructed by a transformer neural network and a binary classifier ,and one-dimensional pitch sequences are pre-trained for the music emotion ,recognition task.

[0110] The above-mentioned conditional control information is input into the improved absorbing state diffusion model. During the training phase of the prediction model, in order to guide the reverse diffusion process to reconstruct the melody prior, the cross-attention module is used to effectively capture the complex dependency between the melody feature sequence and the corresponding emotional condition time series, and learn the interaction pattern between the two. Finally, the conditional control of the single-track melody generation with the emotional information in the time dimension is completed. The pre-trained reverse diffusion part is used as the melody generator, the weights are frozen in subsequent tasks, and then used for fine-tuning of the model. This step specifically includes: based on the self-made EEG and music pairing dataset, the input of the Mamba network is the transformed feature sequence F of the above-mentioned EEG emotion information extraction module E , F E ∈R N×T×Z The output is a feature sequence transformed by the above-mentioned melody emotion extractor, and a mapping from EEG emotion information to music emotion information is obtained; the loss function for evaluating the fine-tuning process is a weighted sum of the music generation quality loss sub-function and the emotion deviation sub-function, and the pre-trained model is fine-tuned using the gradient descent method to reduce the information deviation between different data sets, and finally a mapping from EEG to emotion and then to music is obtained.

[0111] like Figure 4 As shown in the figure, the process of obtaining a small dataset of EEG and music pairing is as follows:

[0112] 1) Before the stimulation, music professors with experience in music arrangement and performance at the conservatory selected music that evoked specific emotions (happiness, sadness, or calmness). The music was played for 48 seconds, avoiding music that could potentially cause emotional fluctuations. Regarding the experimental design, participants were given 48 seconds to listen to the music clips and 10 seconds to rate their emotions.

[0113] 2) The subjects underwent music stimulation experiments in an environment shielded from external noise interference, where they could clearly hear the music content. All subjects received additional music training before the experiment began.

[0114] 3) During the experiment, the subjects first received random music stimulation. After the music stimulation ended, they were evaluated for emotion and their emotions were calmed down.

[0115] 4) Set a rest period, and then repeat this process until the subject completes all the music stimulation processes of the entire paradigm.

[0116] 5) The above experiments only collected a partial dataset. To increase the sample size, we extracted a subset of EEG-music matching data from four previous datasets. The stimuli for these experiments were carefully selected and lasted between 10 and 48 seconds.

[0117] 6) The aggregated dataset contains the neural data of 76 subjects, all of whom are school students with an average age of 22.7±1.8 years. All participants have normal hearing and vision, are in good health, maintain a stable mental state, and have no obvious medical or psychiatric history.

[0118] 7) The above-mentioned EEG signal preprocessing method and symbolic music processing technology are used to extract features from the aggregated data set, and the final data format is shown in Table 2 below.

[0119] Table 2 Format of paired EEG and music datasets after preprocessing

[0120] Data Type Data shape Data content EEG 6742×32×4×8×9 Number of samples × time frame × frequency band × height × width music 6742×128×1 Number of samples × sequence length × pitch value

[0121] This application adopts a time window of set length, and uses domain transformation and music processing technology to perform feature processing on EEG and music data to obtain EEG two-dimensional feature time frame and pitch time series; convolution and long short-term memory fusion network are used on EEG emotional data to obtain a pre-trained EEG to emotion mapping module; conditional absorption state diffusion model is used on emotional music data to obtain a pre-trained emotion to music mapping module; Mamba network is used to fine-tune the pre-trained model on self-made data to complete the mapping from EEG to emotion and then to music, realizing an EEG music generation method that can make up for the lack of data sets, while retaining the high-quality effect of music generation by the diffusion model, enhancing the ability of emotional control, that is, improving the accurate expression of the musical emotion corresponding to the EEG signal. For example: when an EEG signal representing happiness is input, the output music is also pleasant and happy.

[0122] Based on the same inventive concept, the present application also provides an embodiment of a brainwave music generation device for implementing the aforementioned brainwave music generation method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the brainwave music generation device provided below can be found in the above-mentioned limitations of the brainwave music generation method and will not be repeated here.

[0123] In an exemplary embodiment, the present application provides a brainwave music generation device comprising:

[0124] The dataset acquisition module is used to obtain EEG emotion datasets, emotion music datasets, and EEG music pairing datasets.

[0125] The first pre-training module is used to train a fusion network based on the EEG emotion data set to obtain a pre-trained EEG to emotion mapping module; the fusion network is a fusion network including a convolutional neural network and a long short-term memory network.

[0126] The second pre-training module is used to train the improved diffusion model based on the emotional music data set to obtain a pre-trained emotion-to-music mapping module; the improved diffusion model is used to use music emotion as a conditional control to generate music under the conditional control.

[0127] A model fine-tuning module is used to fine-tune the pre-trained model according to the EEG music pairing dataset to obtain an EEG music generation model; the pre-trained model includes the EEG to emotion mapping module, the Mamba network and the emotion to music mapping module; the Mamba network is used to generate music emotion information based on the EEG emotion information generated in the EEG to emotion mapping module, and input the music emotion information into the emotion to music mapping module; the EEG music generation model is used to generate music based on EEG signals.

[0128] In an exemplary embodiment, Figure 5 As shown, the present application provides a brainwave music generation device comprising:

[0129] The data acquisition module is used to collect two large open-source EEG emotion and emotional music datasets, and a self-made small dataset of EEG signals paired with stimulating music; the two large open-source EEG emotion and emotional music datasets include: DEAP dataset, EMOPIA dataset, Lakh dataset, etc.; the self-made small dataset includes: collected EEG signal data and music files that stimulate the brain.

[0130] The data processing module is used to extract features from EEG and music data using a time window of a set length and applying domain transformation methods and symbolic music processing technology to obtain two-dimensional feature time graph frames of EEG and pitch time series of single-track melody; the domain transformation methods include: time domain transformation, frequency domain transformation, and time-frequency domain transformation; the symbolic music processing technology is taken from the music feature processing submodule in MusicVAE proposed by Magenta.

[0131] The EEG emotion information mapping module is used to use a convolution and long-short-term memory fusion network on a large EEG emotion dataset to obtain a pre-trained EEG to emotion mapping module; the fusion network is composed of a convolutional neural network and a long-short-term memory network connected in series; the pre-trained mapping module can convert the input EEG signal into a time series of emotion features.

[0132] The emotional music generation module is used to use an improved conditionally controlled absorbing state diffusion model on a large emotional music dataset to obtain a pre-trained emotion-to-music mapping module; the improved conditionally controlled absorbing state diffusion model uses the emotional information of the music as conditional control and completes the conditionally controlled generation of emotional music through training.

[0133] The emotion bias fine-tuning module is used to fine-tune the pre-trained model on a self-made small dataset using the Mamba network and a specialized loss function to complete the mapping from EEG to emotion and then to music; the loss function is a weighted sum of the music generation quality loss sub-function and the emotion bias sub-function.

[0134] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store brainwave music generation data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for generating brainwave music is implemented.

[0135] Those skilled in the art will understand that Figure 6 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application and does not constitute a limitation on the computer device to which the solution of the present application is applied. A specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of the above-mentioned method embodiments when executing the computer program.

[0136] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0137] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0138] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0139] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0140] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, data processing logic of programmable logic devices, and the like.

[0141] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for generating brainwave music, characterized in that: The brainwave music generation method comprises: Obtain an EEG emotion dataset, an emotional music dataset, and an EEG music pairing dataset; the EEG emotion dataset and the emotional music dataset are open source datasets, and the EEG music pairing dataset is constructed by collecting EEG signals under emotional music stimulation; Training a fusion network based on the EEG emotion dataset to obtain a pre-trained EEG to emotion mapping module; the fusion network is a fusion network including a convolutional neural network and a long short-term memory network; An improved diffusion model is trained based on the emotional music dataset to obtain a pre-trained emotion-to-music mapping module; the improved diffusion model is used to generate music under the conditional control using the musical emotion as a conditional control; According to the EEG music pairing dataset, the pre-trained model is fine-tuned to obtain an EEG music generation model; the pre-trained model includes the EEG to emotion mapping module, the Mamba network and the emotion to music mapping module; the Mamba network is used to generate music emotion information based on the EEG emotion information generated in the EEG to emotion mapping module, and input the music emotion information into the emotion to music mapping module; the EEG music generation model is used to generate music based on EEG signals.

2. The method for generating brainwave music according to claim 1, wherein: The EEG emotion dataset includes the DEAP dataset, and the emotional music dataset includes the EMOPIA dataset and the Lakh dataset.

3. The method for generating brainwave music according to claim 1, wherein: The fusion network is trained based on the EEG emotion dataset to obtain a pre-trained EEG to emotion mapping module, specifically including: Slicing the EEG signal of each sample in the EEG emotion dataset using a time window of a set length, extracting features from the EEG signal of each slice using a domain transformation method, and obtaining a two-dimensional feature time frame of the EEG signal; The fusion network is trained with the two-dimensional feature time image frames of each EEG signal as input and the corresponding EEG emotion as output to obtain a pre-trained EEG-to-emotion mapping module; The fusion network also includes a first classifier. The convolutional neural network in the fusion network is used to extract a first feature sequence including spatial information of the EEG signal from the two-dimensional feature time frame. The long short-term memory network is used to extract a second feature sequence including spatial information and time information of the EEG signal from the first feature sequence. The first classifier is used to perform EEG emotion recognition based on the second feature sequence.

4. The method for generating brainwave music according to claim 3, wherein: The improved diffusion model is trained based on the emotional music dataset to obtain a pre-trained emotion-to-music mapping module, which specifically includes: The symbolic music feature processing module in MusicVAE is used to perform data processing and feature extraction on the music signal of each sample in the emotional music dataset to obtain a one-dimensional pitch time series of a single-track melody; The one-dimensional pitch time series of the single-track melody is sliced ​​using a time window of set duration to obtain a one-dimensional pitch time frame; A transformer neural network is used to capture the global dependencies and contextual information of the pitch time series of a single-track melody, obtaining a feature sequence of musical emotion information. The second classifier is used to perform emotion recognition on the music emotion information feature sequence; The improved diffusion model is trained by taking the music emotion information feature sequence as the conditional control information of the improved diffusion model to obtain a pre-trained emotion-to-music mapping module.

5. The method for generating brainwave music according to claim 4, wherein: Based on the EEG music pairing dataset, the pre-trained model is fine-tuned to obtain the EEG music generation model, which specifically includes: During fine-tuning of the pre-trained model according to the EEG music pairing dataset, the second feature sequence output by the long and short-term memory network in the EEG to emotion mapping module is input into the Mamba network, and the music emotion information feature sequence output by the Mamba network is input into the emotion to music mapping module.

6. The method for generating brainwave music according to claim 4, wherein: The improved diffusion model adds a cross-attention module between the multi-head self-attention layer and the feedforward neural network in the transformer block of the original diffusion model, wherein the query vector in the cross-attention module inputs the melody sequence information output by the multi-head self-attention layer, and the key vector and value vector in the cross-attention module input the music emotion information feature sequence.

7. The method for generating brainwave music according to claim 1, wherein: According to the EEG music pairing dataset, the loss function in the process of fine-tuning the pre-trained model is expressed as: L t =λ1L rf +λ2L rvb ; Among them, L t is the overall loss function value, λ1 and λ2 are weight parameters, L rf For emotional loss, L rvb Loss of music quality.

8. A device for generating brainwave music, characterized in that: The brainwave music generating device comprises: The dataset acquisition module is used to obtain EEG emotion datasets, emotion music datasets, and EEG music pairing datasets; A first pre-training module is used to train a fusion network based on the EEG emotion dataset to obtain a pre-trained EEG to emotion mapping module; the fusion network is a fusion network including a convolutional neural network and a long short-term memory network; A second pre-training module is used to train an improved diffusion model based on the emotional music dataset to obtain a pre-trained emotion-to-music mapping module; the improved diffusion model is used to use music emotions as conditional control to generate music under the conditional control; A model fine-tuning module is used to fine-tune the pre-trained model according to the EEG music pairing dataset to obtain an EEG music generation model; the pre-trained model includes the EEG to emotion mapping module, the Mamba network and the emotion to music mapping module; the Mamba network is used to generate music emotion information based on the EEG emotion information generated in the EEG to emotion mapping module, and input the music emotion information into the emotion to music mapping module; the EEG music generation model is used to generate music based on EEG signals.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for generating brainwave music according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating brainwave music according to any one of claims 1 to 7 is implemented.

Images