Systems and methods for decoding chord information from brain activity

Abstract

Disclosed are a system and method for decoding chord information from brain activity, comprising: obtaining raw data of brain activity of one or more subjects while they listen to music labeled with chord labels; extracting brain activity patterns from the raw brain activity data; temporally coupling the brain activity patterns and music data to form training data for a decoding model; training the decoding model; optionally, fine-tuning the trained decoding model by labeling and using a small amount of originally unlabeled brain activity data; obtaining a second batch of raw brain activity from the subjects via functional neuroimaging during various psychological music activities; and mapping the second batch of brain activity to corresponding chord information.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application Serial No. 63 / 106,486, filed on October 28, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application discloses systems and methods for decoding chord information from brain activity. Background Art

[0004] A chord is a combination of multiple pitches that sound as if they were played simultaneously. Chords and their sequences (chord progressions) can largely determine the emotional interpretation of music, evoking specific subjective feelings and are therefore crucial to music perception and most music creation processes. In the field of music information retrieval, a large amount of research is devoted to achieving better automatic chord estimation (ACE) - which is considered one of the most important tasks in the field.

[0005] In addition to extracting chords from existing music, in some scenarios, people may also want to extract chords from inner music (such as musical memory, musical imagination, musical hallucinations, earworms, etc.), for example, recording chord progressions in the process of music creation, understanding the emotional valence of inner music stimulation for medical care, etc. However, in this case, since only subjective music experience but no objective audio signal is available, the traditional ACE method cannot achieve chord extraction of inner music.

[0006] Neuropsychological studies have shown that music perception and imagination have similar neuronal mechanisms and can cause similar brain activation patterns. There are currently some neuroscience studies dedicated to reconstructing music stimuli from brain activity in music listening and music imagination. However, when it comes to chord information, the above-mentioned stimulus reconstruction-based techniques will greatly limit the accuracy of chord estimation due to the limited reconstruction accuracy of full stimulus decoding. In addition, the process from music transcription to chords will also lead to additional information loss. Therefore, in order to overcome the above limitations, the inventors of the present invention propose to directly estimate chord information from brain activity. Summary of the invention

[0007] In order to provide a basic understanding of some aspects of the present invention, a brief summary of the present invention is given below. This summary is not a broad overview of the present invention. It is neither intended to identify the key or main elements of the present invention, nor is it intended to describe the scope of the present invention. On the contrary, the only purpose of this summary is to present some concepts of the present invention in a simplified form as a prelude to the more detailed description presented below.

[0008] There is currently no direct method to decode chord information from brain activity. One possible approach is to first reconstruct the music stimulus using existing auditory stimulus decoding technology, and then use automatic chord estimation technology to estimate the chord information from the reconstructed music. However, severe information loss occurs when reconstructing auditory stimuli using existing neural decoding technology, and the subsequent automatic chord estimation also causes secondary information loss.

[0009] Reading chord information from the brain has wide applications in multiple fields, such as healthcare for mental illness and music composition. However, there is currently no available technology that can accomplish such a task. Current methods (such as reconstructing musical stimuli) have low accuracy and can easily lose chord information during the reconstruction process.

[0010] In the present invention, these problems are addressed by using a deep learning-based approach to decode chord information directly from brain activity.

[0011] On the one hand, the present application provides a method for training a decoding model for decoding chord information from brain activity, comprising: obtaining raw brain activity data from a subject while the subject listens to music marked with chord labels; extracting brain activity patterns from the raw brain activity data; temporally coupling the extracted brain activity patterns and corresponding chord labels to form training data for the decoding model; and training the decoding model based on the training data, wherein the input of the decoding model is the brain activity pattern and the output is chord information. The chord information may include a combination of one or more of the following information: chord type, root note, or other chord-related information.

[0012] On the other hand, the present application provides a method for decoding chord information from brain activity, comprising: acquiring raw brain activity data of a subject via functional neuroimaging during various psychological music activities; extracting brain activity patterns from the raw brain activity data; and providing the extracted brain activity patterns as input to a trained decoding model for decoding chord information from brain activity to obtain corresponding chord information, wherein the decoding model is obtained according to the above-mentioned training method.

[0013] On the other hand, the present invention describes a system for transcribing, generating and recording chords, comprising: a memory storing functional units and a processor executing the functional units stored in the memory, wherein the functional units include: (1) a learning module, comprising: a functional neuroimaging component for measuring brain activity of a subject during listening to music labeled with chords, a signal processing component for extracting brain activity patterns, a well-defined database of music-related chord labels, and a decoding model for training with a predefined architecture; and (2) a decoding module, comprising: a functional neuroimaging component for measuring raw brain activity in various psychological music activities, a signal processing component for extracting brain activity patterns suitable for input, a trained decoding model derived from the learning module for converting input data into chord information, and a data output component configured to output chord information from the trained decoding model.

[0014] On the other hand, this article describes a method for decoding chord information from brain activity, comprising: obtaining raw data of brain activity of one or more subjects while they listen to music labeled with chord labels; extracting brain activity patterns from the raw brain activity data; temporally coupling the brain activity patterns and music data to form training data for a decoding model; training the decoding model; optionally, fine-tuning the trained decoding model by labeling and using a small amount of originally unlabeled brain activity data; obtaining a second batch of raw brain activity from the subjects via functional neuroimaging during various psychological music activities; and mapping the second batch of brain activity to corresponding chord information.

[0015] On the other hand, this article describes a system for a chord decoding protocol, comprising: a memory for storing functional units and a processor for executing the functional units stored in the memory, wherein the functional units include: (1) a neural code extraction model, whose function is to generate raw data from existing music-related neuroimaging datasets and offline neural measurements of users when listening to music, and then extract processed brain activity patterns / features (i.e., chord-related neural codes) from the raw data; (2) a decoding model, whose function is to learn and estimate an approximate mapping from neural codes to inner musical chords; and (3) a trained model, whose function is to convert neural codes in an application scenario into estimated values ​​of chord information and can perform fine-tuning operations.

[0016] To achieve the foregoing and related ends, the present invention comprises the features hereinafter fully described and particularly pointed out in the claims. The following description and drawings set forth in detail certain illustrative aspects and embodiments of the present invention, however, these contents are exemplary-that is, they refer only to several possible ways among many ways to employ the principles of the present invention, and are not exhaustive descriptions. Other objects, advantages and novel features of the present invention will become apparent from the following detailed description of the invention when considered in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a method for training a decoding model for decoding chord information from brain activities according to one aspect of the present invention is shown.

[0018] Figure 2 Depicted is a schematic diagram of a pipeline for decoding chord information from brain activity in accordance with one aspect of the present subject matter.

[0019] Figure 3 An embodiment of an example of an architecture for a decoding model is depicted.

[0020] Figure 4 An embodiment of a flow diagram depicting the computational process in a learning and decoding module.

[0021] Figure 5 Table 1 is shown, which reports experimental results compared with related prior arts.

[0022] Figure 6 A block diagram of an example electronic computing environment is shown that can be implemented in conjunction with one or more aspects described herein.

[0023] Figure 7 Depicted is a block diagram of an example data communications network that can operate in conjunction with various aspects described herein. DETAILED DESCRIPTION

[0024] The subject matter described in the present invention can be simply understood as "brain reading" focusing on decoding chord information in music listening, music imagination or other psychological processes. Chord information extraction is usually based on the music fragment itself, and has never been achieved by a computational method based on neuroscience. In the present disclosure, the inventors describe a new method for decoding chord information from brain activity. The specific problems solved by the present invention include but are not limited to: 1) In clinical scenarios, the assessment of auditory hallucination symptoms usually relies on a self-report system, which lacks accuracy, and the system and method described herein can help doctors and medical staff better understand the nature of the inner voice of patients with musical hallucination (MH) and musical ear syndrome (MES) to improve the quality of treatment and medical care. The above-mentioned intelligent medical care system for MH and MES patients is also considered novel. 2) For music lovers and creators, manual processing of chords may be laborious and even lead to interruptions in the creative process, and the system and method described in the present invention can provide a more efficient and convenient way to transcribe, generate and record chords and chord progressions from their subjective perception or cognitive experience without requiring them to invoke motor functions (such as humming, reciting or notating). The above-mentioned intelligent systems for transcribing, generating and recording chords are also considered novel.

[0025] The human brain has evolved computational mechanisms to convert musical stimuli into high-level information such as chords. Even for subjects who have not received musical training, important chord information such as chord quality (i.e., chord type) can still be perceived consciously or unconsciously by them and thus embedded in their brain activity. In the present disclosure, a new method for decoding chord information from brain activity is described. Various aspects of the method include acquiring and processing brain activity data from subjects or users, training a decoding model using labeled brain activity and music data, fine-tuning the trained decoding model by labeling and using a small amount of originally unlabeled brain activity, and mapping the brain activity to corresponding chord information.

[0026] Figure 1 A training method for a decoding model for decoding chord information from brain activity according to an embodiment of the present invention is shown, which includes: S1) obtaining original brain activity data from one or more subjects while the one or more subjects listen to music marked with chord labels; S2) extracting brain activity patterns from the original brain activity data; S3) establishing a correspondence between the extracted brain activity patterns and corresponding chord labels to form training data for the decoding model; S4) training the decoding model based on the training data, wherein the input of the decoding model is the brain activity pattern and the output is the chord information.

[0027] Wherein, in step S1), when the subject listens to music with chord labels, raw brain activity data is obtained from the subject through functional neuroimaging. The music with chord labels here can be music that is pre-labeled with chord information such as root note and chord type. The raw brain activity here refers to the measurement of brain activity using any type of functional neuroimaging technology, which may include but is not necessarily limited to functional magnetic resonance imaging (fMRI), functional near-infrared spectroscopy (fNIRS), electroencephalogram (EEG), magnetoencephalogram (MEG), functional ultrasound imaging (fUS) and positron emission tomography (PET). When invasive recording is available, electrocorticogram (ECoG) and intracortical recording (ICoR) are also included.

[0028] In step S2), the acquired raw brain activity is processed and brain activity patterns are extracted. The processing of raw brain activity may vary for different neuroimaging methods, but it should generally include the steps of preprocessing, region of interest (ROI) definition, and brain activity pattern extraction. In cases where voxel-by-voxel analysis is more appropriate, the definition of ROI should be all voxels. For three-dimensional data (such as fMRI data), the raw data is encoded using spatial information. For two-dimensional data (such as EEG / MEG data), the raw data is encoded using channel information, and source reconstruction may be considered before feeding the data to the learning and decoding module. The nature of brain activity patterns may vary due to the different temporal resolutions of different neuroimaging methods. For data with lower temporal resolution (such as fMRI data), spatial patterns (i.e., the distribution of brain activity across ROI) may be considered. For data with high temporal resolution (such as EEG / MEG data), spatiotemporal patterns may be considered.

[0029] In step S3), a corresponding correspondence is established between the extracted brain activity patterns of the subject and the chord labels of the music listened to by the subject to form training data. For example, the extracted brain activity patterns and corresponding chord labels are coupled in time to form training data for the decoding model. The chord labels and brain activity patterns are coupled in time. The decoding model can be a deep neural network (or any other type of computational model for the same purpose, such as a support vector machine or other machine learning model), and its architecture can vary over a wide range, which can include but is not limited to dense neural networks, spatial or spatiotemporal convolutional neural networks (CNNs) and recurrent neural networks (RNNs). Generally, when spatial patterns are applied, dense neural networks can be considered.

[0030] In step S4), the decoding model is trained based on the formed training data, and the decoding model is trained until convergence. The hyperparameters of the decoding model can be adjusted by cross-validation. The input of the decoding model is the brain activity pattern, and the output is the chord information. The output chord information corresponds to the chord label. In one example, the output chord information may include the root note and / or the chord type. When considering slash chords (compound chords and inversion chords), bass may also be included.

[0031] After the trained decoding model is obtained through the above steps, the decoding model can be used to decode the corresponding chord information from the brain activity of any subject or user. In one embodiment, the method for decoding chord information from brain activity may include obtaining the original brain activity data of the subject or user to be decoded from a wide range of psychological music activities through functional neuroimaging (for example, in a manner similar to step S1); then similar to step S2, extracting brain activity patterns from the original brain activity data; then the extracted brain activity patterns can be provided as input to the trained decoding model to obtain the corresponding chord information. The psychological activities here are such as music listening, music hallucination, music imagination or synesthesia (for example, visual imagination that may evoke music experience). When the properties of the data obtained in various psychological music activities are different from the properties of the data in the training decoding model, the unlabeled brain activity can also be used to fine-tune the trained decoding model. For example, the parameters of the lower layer of the decoding model can be fixed, and the normal training steps can be performed to adjust the parameters of the upper layer of the model.

[0032] refer to Figure 2 , which shows the overall pipeline of the system and method described herein. It consists of a learning module and a decoding module. The general steps / actions are as follows. It is not necessary to perform every step / action in every case. The various aspects and objectives described herein can be achieved by performing a subset of the following steps / actions.

[0033] One step / action is to obtain raw brain activity from the subject by functional neuroimaging while the subject listens to music with chord labels. Raw brain activity here refers to the measurement of brain activity using any type of functional neuroimaging technology, which can include but is not necessarily limited to functional magnetic resonance imaging (fMRI), functional near-infrared spectroscopy (fNIRS), electroencephalography (EEG), magnetoencephalography (MEG), functional ultrasound imaging (fUS), and positron emission tomography (PET). Where invasive recordings are available, electrocorticography (ECoG) and intracortical recordings (ICoR) are also included.

[0034] Another step / action is to process the raw brain activity and extract brain activity patterns. The processing of raw brain activity may vary for different neuroimaging modalities, but it should generally include the steps of preprocessing, region of interest (ROI) definition, and brain activity pattern extraction. In cases where voxel-by-voxel analysis is more appropriate, the definition of the ROI should be all voxels. For three-dimensional data (such as fMRI data), the raw data is encoded using spatial information. For two-dimensional data (such as EEG / MEG data), the raw data is encoded using channel information, and source reconstruction is preferably performed before feeding the data to the learning and decoding module. The nature of brain activity patterns may vary due to the different temporal resolutions of different neuroimaging modalities. For data with lower temporal resolution (such as fMRI data), spatial patterns (i.e., the distribution of brain activity across ROIs) may be considered. For data with high temporal resolution (such as EEG / MEG data), spatiotemporal patterns may be considered.

[0035] Another step / action is to pass the brain activity patterns and chord labels to the decoding model. The chord labels and brain activity patterns are temporally coupled to each other. The decoding model is a deep neural network (or any other type of computational model used for the same purpose, such as a support vector machine or other machine learning model), while its architecture can vary widely, and can include but is not limited to dense neural networks, spatial or spatiotemporal convolutional neural networks (CNNs), and recurrent neural networks (RNNs). Typically, when spatial patterns are applied, dense neural networks are considered. The decoding model takes brain activity patterns as input and chord labels as output.

[0036] Another step / action is to train the decoding model until convergence. The hyperparameters of the model can be tuned through cross-validation.

[0037] Another step / action is to save the trained model and load it into the decoding module.

[0038] Another step / action is, if the decoding module requires, to fine-tune the decoding model using the data from the decoding module after manually labeling them and return to the saving step / action.

[0039] Another step / action is to obtain raw brain activity from the user through functional neuroimaging in a wide range of psychological activities, such as music listening, music hallucination, music imagination, or synesthesia (e.g., visual imagination that may evoke musical experience). When the nature of the acquired data is different from that in the first step, a learning module is required to fine-tune the decoding model.

[0040] Another step / action is to process the raw brain activity and extract the brain activity pattern, which is the same as the aforementioned step / action of processing the raw brain activity and extracting the brain activity pattern.

[0041] Another step / action is to pass the brain activity pattern to the decoding model and output the chord information. Depending on the specific task, the output can include the root note and chord type; when considering slash chords (compound chords and inversions), it can also include the bass. After that, the decoded chord information can be passed and used for specific application scenarios, such as healthcare or music creation.

[0042] Common apparatus for chord decoding include a computer or any other type of programmable executable processor capable of performing all of the data input, processing, and output steps of the method.

[0043] This article describes systems and methods for decoding chord information directly from brain activity rather than music. The systems and methods described herein overcome the limitations of traditional ACE methods in processing inner music and can improve the quality of specific medical care, music creation, etc.

[0044] Example

[0045] The present invention can be understood through an operational embodiment. In terms of results, the three best performing subjects achieved accuracies of 98.5%, 97.9% and 96.8% in the chord type decoding task, and 93.0%, 88.7% and 84.5% in the chord decoding task. Since natural music was used in this experiment, these results show that the method is accurate and robust to fluctuations in non-chord factors.

[0046] Original use of the dataset: The dataset used in this example comes from a previous study [SAARI, Pasi et al., Decoding musical training from dynamic processing of musical features in the brain. Scientific reports, 2018, 8.1: 1-12.]. The main purpose of this previous study was to distinguish whether a subject had received music training based solely on his / her fMRI signal during music listening. Musical stimuli and fMRI signals were provided in this previous study.

[0047] Chord marking: The transcription of music to chords is one of the basic training for musicians. The inventors manually marked the chords of the musical stimuli with the help of professional musicians to obtain the chord information.

[0048] step:

[0049] First, 36 subjects (including 18 musicians and 18 non-musicians) were recorded using a 3T scanner while listening to music stimuli. 80% and 10% of the data were used for training and cross-validation in the learning module, and 10% of the data were used for testing in the decoding module. Only major and minor triads were considered in this experiment.

[0050] Second, the recorded fMRI data were realigned, spatially normalized, artifact minimized, and detrended using the Statistical Parametric Mapping Toolbox. Automated Anatomical Labeling 116 (AAL-116) was used for ROI definition. The mean of all signals within each partition at each time point was applied to generate the spatial pattern.

[0051] Third, the brain activity patterns and chord labels are passed to the decoding model. Figure 3 An example of the architecture of the decoding model is shown, which is a dense neural network with 5 hidden layers. The spatial distribution of 116 ROIs is taken as input. The output layer consists of 13 units. The first unit represents the chord type (0 for minor chords and 1 for major chords). For the other 12 units, normalized exponential processing (softmax) and one-hot encoding are applied, and each of these units represents a root note, namely C, C#, D, D#, E, F, F#, G, G#, A, A#, and B.

[0052] Fourth, the decoding model is trained until convergence. Stochastic gradient descent is used for optimization, and dropout regularization is applied to prevent overfitting.

[0053] Fifth, save the trained model and load it into the decoding module.

[0054] Sixth, skip this step because the nature of the data in the learning module and the decoding module is the same and no fine-tuning is required.

[0055] In steps 7 to 9, the test data is processed using the same method as in step 2 and then passed to the trained decoding model. The chord information is output.

[0056] Mathematical description of the general chord decoding protocol

[0057] The following basic symbols are used:

[0058] f e Neural Code Extraction Model

[0059] M α Raw brain activity measurements (for model training)

[0060] X α Neural encoding (for model training)

[0061] Y α Chord labels (for model training)

[0062] Chord labels (for model training)

[0063] L α Training loss

[0064] M β Raw brain activity measures (for model validation)

[0065] X β Neural encoding (for model validation)

[0066] Y β Chord labels (for model validation)

[0067] Estimated chord information (for model validation)

[0068] L β Validation loss

[0069] f d Decoding Model

[0070] M Raw brain activity measurements (for applications)

[0071] X-Neural Coding (for applications)

[0072] Y Chord Label (for application)

[0073] Estimated chord information (for applications).

[0074] In one embodiment, the process involves three main computational operations:

[0075] (1) Extraction of neural code,

[0076] (2) Development of decoding models, and

[0077] (3) Deployment of the trained model (i.e., estimation of chords).

[0078] The flowchart of the three parts of the calculation process in the learning module and the decoding module is Figure 4 Demonstrated and explained in. Figure 4 The details are further explained in the following sections.

[0079] 1) Extraction of Neural Codes

[0080] Original functional neuroimaging measures

[0081] The raw online functional neuroimaging measurements from a specific spatial location s of the signal source during a specific time point t are denoted as M(t, s). Note that s can be in different formats for different neuroimaging modalities. For example, for EEG / MEG, s refers to the number of electrodes / channels n or the 2D scalp coordinate values ​​{x, y}, while for EEG / MEG with source reconstruction or fMRI, s refers to the 3D spatial coordinate values ​​of the voxel {x, y, z}.

[0082] For model training and validation, raw data is derived from existing music neuroimaging datasets and / or offline measurements of users taken during music listening (i.e., neuroimaging databases), the latter of which can be considered for fine-tuning models developed based on the former. Chord labels for the music used in these listening tasks are obtained and associated with corresponding brain activity measurements. In a holdout validation setting, in one embodiment, these data (raw brain activity measurements with chord labels) are randomly divided into training data {M α , Y α} and verification data {M β , Y β}, the ratio is |M α |∶|M β |=r:1 (usually r=8, where |A| refers to the number of elements in set A). In another embodiment (in a cross-validation setting), the data will be randomly divided into r+1 subgroups. The learning can be repeated r+1 times. In each repetition, each subgroup is used for validation and the other r subgroups are used for training.

[0083] General format of neural codes

[0084] The term neural code (X) in this paper refers to the processed brain activity patterns / features extracted from raw functional neuroimaging measurements M during music-related psychological processes (e.g., music listening, imagination, hallucination), which are the real inputs to the decoding model. e is an empirical, deterministic function that maps M to X through a series of signal processing operations, which can be accomplished using standard neuroimaging processing tools (e.g., Statistical Parameter Mapping Toolbox, EEGLAB, FieldTrip Toolbox). e The specific form of f varies with different neuroimaging methods. In principle, f e This includes preprocessing (e.g., filtering, normalization, artifact removal, correction) and spatial averaging of the signal over each region of interest (ROI). Channel-based source reconstruction of neuroimaging data is optional but often used in practice. eThe overall goal of applying M is to improve the quality of brain activity signals and enhance their coupling with chord information. When the raw measurements are used directly as features of interest (i.e., X = M), f e degenerates into the identity map f I : A→A. At each time point, the elements in the input X are the distribution of activation values ​​of all ROIs in the brain (e.g., for a 116-ROI study, at each time point, the input has a vector {x1, x2, …, x 116} form).

[0085] For training and validation data M α and M β , N α and N β According to X α =f e (M α ), X β =f e (M β ) is obtained. In one embodiment, note that M α and M β It is recommended that data be acquired during music listening (rather than musical imagination, musical hallucination, or synesthesia) to ensure controllability of the chord label Y.

[0086] 2) Development of decoding model

[0087] Description of the chord decoding problem

[0088] The chord decoding problem is to estimate the mapping between the neural code X and the chord Y of the inner music, that is, to generate a decoding model f from X and Y d .

[0089] Y is the output of the decoding model; each element in the output Y includes the root note and the chord type, the latter of which carries information about the emotional valence; each sample in Y is represented as a one-hot encoded representation (for example, when considering 48 major, minor, diminished, and augmented triads, the chord "C minor" can be represented as If the chord type considered is binary, e.g. major / minor, the chord type representation can be further compressed to a single bit).

[0090] In one embodiment, the methods described herein do not require reconstruction of the music clip.

[0091] Learning model selection

[0092] Depending on the nature of the neuroimaging modality and the availability of computational resources, different computational models can be applied, including but not limited to dense neural networks, spatial convolutional neural networks (spatial CNNs), spatiotemporal convolutional neural networks (spatiotemporal CNNs), and recurrent neural networks (RNNs).

[0093] Typically, dense neural networks are used when each sample represents a single temporal data point with a distribution of activation values ​​across all ROIs in the brain (however, other architectures can also be used). For each hidden layer, the node value where g(·) is the activation function, is the i-th node in layer k, is the jth node in layer k+1, is the corresponding weight, and b is the bias. Typically, a linear rectifier unit is used as the activation function, i.e. There should be a softmax layer after these layers where z (root，type)i is the i-th in the last layer. By considering the spatial information of ROI, spatial CNN is also commonly used for this data structure and also has good performance.

[0094] For data structures where each sample represents a series of temporal data points, spatiotemporal CNNs and RNNs can be used, and additional temporal information can be provided and utilized. However, such data structures may lead to difficult temporal grouping / segmentation problems (i.e., one sample may cover multiple chords), so they are not recommended unless special attention is paid to this problem.

[0095] Decoding accuracy and loss function

[0096] The decoding accuracy is defined as where t T is the total duration correctly estimated, t F The total duration of the incorrect estimate is given by . The cross entropy loss for training and validation satisfies Where (y (root) ) i is the i-th value of the root note label, is the i-th value of the softmax output of the root note label, (y (type) ) i is the i-th value of the chord type label, is the i-th value of the softmax output of the chord type label. For healthcare applications, perhaps only the chord type is of interest, where

[0097] Training (fitting) and validation

[0098] During the training phase, the decoding model f d The parameters of are first randomly initialized and then updated via back-propagation. There are a variety of back-propagation algorithms available (e.g., stochastic gradient descent, Adam), and they can be easily implemented using standard deep learning packages. Dropout regularization can optionally be applied to avoid overfitting.

[0099] Cross-validation or leave-out validation can be performed to further adjust f d Hyperparameters (e.g. model architecture, learning rate).

[0100] 3) Deployment of trained models

[0101] Reasoning (Decoding)

[0102] Then the trained model f d Applied to the user's neural code X to obtain an estimate of the chord information This solves the chord decoding problem defined in the chord decoding problem description section.

[0103] Fine-tuning

[0104] When the neuroimaging measurements from the user are highly heterogeneous with those used to train the decoding model, the decoding module sends instructions to the learning module to fine-tune the operation. Specifically, as Figure 3 As shown, the trained f d The lower-level parameters of the model (such as the weights and biases of the last two layers, corresponding to the high-level parameters of the model) are fixed; then a small amount of neuroimaging measurements from the user are input into the model. In this case, a small number of chords need to be manually labeled (i.e., Y); then the upper-level parameters (corresponding to the low-level parameters of the model) are updated via backpropagation. The update method can refer to the aforementioned "training (fitting) and validation" steps.

[0105] Results and discussion

[0106] Example decoding model performance

[0107] Leave-one-out cross-validation was performed for each subject to evaluate the cross-subjects. The Top-1 accuracy of the top 3 subjects reached 98.5%, 97.9%, and 96.8% in the chord type decoding task, and 93.0%, 88.7%, and 84.5% in the chord decoding task, respectively. The overall Top-1 accuracy was found to be 88.8% in the chord type decoding task (90.8% for musicians and 86.7% for non-musicians, both significantly above chance level). In the chord decoding task, the overall Top-3 accuracy was found to be 80.9% (95.7% for musicians and 66.1% for non-musicians, both significantly above chance level), and the overall Top-1 accuracy was 48.8% (66.5% for musicians and 31.1% for non-musicians, both significantly above chance level). These results confirm that sufficient information has been encoded in brain activity to decode chord information. In addition, since natural music was used in this experiment, these results also show that the method is accurate and robust to fluctuations in non-chord factors.

[0108] Comparison with related prior art

[0109] Although there is no available technology to decode chord information directly from neural activity except for the present invention, some studies have done similar work by trying to reconstruct musical stimuli or identify musical stimuli from a pool of known brain music fragments. Once the stimulus is reconstructed or identified, ACE can be used to estimate chord information. However, the accuracy of chord information estimation performed by this method is inevitably lower than the accuracy of music reconstruction, because the chord information is estimated based on the reconstructed music. Figure 5 The comparison between the technology described in the present invention and other current technologies is summarized in the following (Table 1).

[0110] Novelty and significance

[0111] This application describes how to decode chord information from brain activity rather than the music itself. It overcomes the limitations of the traditional ACE method in processing inner music and can improve the quality of specific medical care, music creation, etc.

[0112] application

[0113] The present invention can be used as a brain-computer interface (BCI) or provide decoding services for BCI. Potential products and applications of the present invention include many categories:

[0114] Smart healthcare systems for patients with musical hallucinations and musical ear syndrome;

[0115] An imagination-based chord progression generation system for music creators;

[0116] an automatic chord notation system for professional musicians; and

[0117] Entertainment products that translate the user’s brain activity into corresponding chords of their subjective experience.

[0118] There are many applications in the healthcare sector, for example, addressing Musical Ear Syndrome. Musical Ear Syndrome (MES) is described as a non-psychiatric disorder characterized by the perception of music in the absence of external acoustic stimulation. It is reported to affect approximately 5% of the population. It can affect people of all ages with different hearing abilities (including normal hearing, tinnitus, or hearing loss). Because the nature of MES is unknown, the treatment of MES depends largely on the individual case. In some cases, medications can help relieve symptoms, but there is limited evidence to support the prescription of medications for MES. Other treatments for MES may include self-affirmation, such as meditation and distraction.

[0119] According to various case reports, the experience of MES patients can vary significantly. Some patients experience no distress and even find it occasionally enjoyable and interesting, while others find it very annoying or unbearable. This different experience may be caused by different emotional notes of their inner music, which depends largely on the type of chord. These effects may not appear in real time, but days or weeks after the first inner voice appears, which means that early control and prevention are possible. Moreover, at present, the understanding of this impact on patients relies heavily on self-reports.

[0120] The present invention can provide an intelligent healthcare system for MES patients, which helps to objectively identify the chord types of their inner voices, which contain emotional valence information, thereby providing them with better health care and treatment before serious symptoms occur (for example, anti-depressant treatment for patients who frequently experience minor chords or dissonant chords).

[0121] Another healthcare example is musical hallucinations. Musical hallucinations (MH) are a psychopathological disorder in which music is perceived without a source and account for a large portion of auditory hallucinations. It accounts for about 0.16% of total hospitalizations. In elderly subjects with hearing problems, the incidence of musical hallucinations is 2.5%. There is no clear treatment for MH patients. Current treatments are aimed at treating the known underlying causes, such as psychiatric disorders, brain damage, etc. In healthcare, it is necessary to understand the patient's symptoms and their severity.

[0122] Similar to MES, different natures of inner music may have different effects on disease progression. In addition, since MH is a psychopathological disease, some patients may not be able to properly communicate and describe the nature of their inner voices. The present invention can provide MH patients with an intelligent health care system that helps to better understand the emotional valence of their inner voices, thereby providing them with better health care and treatment before the disease worsens.

[0123] Another healthcare example is earworms. Earworms refer to involuntary musical imagery and are common in the general population. It is a common phenomenon experienced by more than 90% of people at least once a week. Earworms should be distinguished from MH, in which the patient perceives the source of the sound as external.

[0124] Earworms are usually harmless, but frequent and continuous exposure to music with certain specific chords can bother people, change their quality of life, and may even lead to mental illness. In addition, people with earworms may be interested in outputting chord progressions for entertainment. The present invention can allow people to monitor the chords of their earworms and better understand their emotional valence to maintain mental health and prevent possible adverse consequences. The present invention can also enable people to better understand their earworms by outputting their chord progressions for entertainment purposes.

[0125] There are many applications for recording inner chords in music creation. For example, there are many applications for recording inner chords. Creating chord progressions is a key step in most music creation. Traditional methods for recording chords may include writing down or humming a melody or chord progression. However, the act of recording can often interfere with the subsequent creative process. In addition, there are some creators who have no problem listening to, imagining, and creating music, but cannot accurately sing it.

[0126] The present invention may provide music composers with a new way to create (including retrieving chord progressions from memory) by simply imagining the chord progressions in their minds without interrupting their creative process.

[0127] Another example of music creation is automatic chord transcription (for professional musicians). Chord transcription is a laborious task. Due to the high time and labor costs, the price of hiring professional musicians to do the notation is also correspondingly high.

[0128] For trained musicians, the present invention can provide them with an automatic way to transcribe new chords by focusing only on the chords of the music without involving their motor system (e.g. notation, humming). Non-musicians can also benefit because the cost of hiring professional musicians to do this work may be reduced because less work is required after applying the present invention.

[0129] Another example of music creation is chord generation based on synesthesia. Many music creators struggle to come up with suitable chord progressions for a specific theme. For example, writing a chord progression about glaciers. There are some applications for generating chord progressions, such as Autochords and ChordChord. However, the chord progressions generated by these applications are usually random or based on existing chord progressions, and therefore either cliché or irrelevant to the given theme.

[0130] The present invention can provide music creators with the ability to convert experiences with other sensory modalities (such as vision) into chord progressions that are similar in subjective experience. For example, the brain activity when seeing a glacier is transmitted to the trained model, and the corresponding chords are obtained. They can use the generated chords for direct creation or as a source of inspiration.

[0131] Sample computing environment

[0132] As mentioned above, the techniques described herein can advantageously be applied to any device and / or network that performs data analysis. Figure 6 The following generic remote computer described in is only an example, and the disclosed subject matter can be implemented by any client with network / bus interoperability and interactivity. Therefore, the disclosed subject matter can be implemented in the environment of a network hosting service, where few or minimal client resources are involved, for example, a network environment where the client device only serves as an interface to the network / bus, such as an object placed in an appliance.

[0133] Although not required, certain aspects of the disclosed subject matter may be implemented in part via an operating system, for use by developers of services for devices or objects, and / or included in application software that operates in conjunction with components of the disclosed subject matter. Software may be described in the general context of computer-executable instructions (e.g., program modules or components) executed by one or more computers (e.g., projection display devices, viewing devices, or other devices). Those skilled in the art will appreciate that the disclosed subject matter may be practiced with other computer system configurations and protocols.

[0134] therefore, Figure 6 An example of a suitable computing system environment 1100 in which some aspects of the disclosed subject matter may be implemented is shown, although as noted above, the computing system environment 1100 is merely one example of a suitable computing environment for an apparatus and is not intended to suggest any limitation as to the scope of use or functionality of the disclosed subject matter. Neither should the computing environment 1100 be interpreted as having any dependency or requirement relating to any one or combination of components shown in the exemplary operating environment 1100.

[0135] refer to Figure 6, an exemplary apparatus for implementing the disclosed subject matter includes a general purpose computing device in the form of a computer 1110. Components of the computer 1110 may include, but are not limited to, a processing unit 1120, a system memory 1130, and a system bus 1121 that couples various system components including the system memory to the processing unit 1120. The system bus 1121 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures.

[0136] Computer 1110 typically includes various computer-readable media. Computer-readable media can be any available media that can be accessed by computer 1110. As an example and not limitation, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technology, CDROM, digital versatile disk (DVD) or other optical disk storage devices, cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other media that can be used to store the desired information and can be accessed by computer 1110. Communication media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal, such as a carrier wave or other transmission mechanism, and includes any information delivery media.

[0137] The system memory 1130 may include computer storage media in the form of volatile and / or nonvolatile memory, such as read-only memory (ROM) and / or random access memory (RAM). A basic input / output system (BIOS), which contains the basic routines that help transfer information between elements within the computer 1110 (e.g., during startup), may be stored in the memory 1130. The memory 1130 also typically contains data and / or program modules that are immediately accessible and / or currently being operated on by the processing unit 1120. By way of example, and not limitation, the memory 1130 may also include an operating system, application programs, other program modules, and program data.

[0138] The computer 1110 may also include other removable / non-removable, volatile / non-volatile computer storage media. For example, the computer 1110 may include a hard disk drive that reads from or writes to a non-removable, non-volatile magnetic medium, a disk drive that reads from or writes to a removable non-volatile disk, and / or an optical drive that reads or writes to a removable non-volatile optical disk, such as a CD-ROM or other optical media. Other removable / non-removable, volatile / non-volatile computer storage media that may be used in the exemplary operating environment include, but are not limited to, magnetic cassettes, flash memory cards, digital versatile disks, digital video tapes, solid-state RAM, solid-state ROM, and the like. A hard disk drive is typically connected to the system bus 1121 via a non-removable memory interface (such as an interface), while a disk drive or optical drive is typically connected to the system bus 1121 via a removable memory interface (such as an interface).

[0139] A user may enter commands and information into the computer 1110 through input devices such as a keyboard and a pointing device, commonly referred to as a mouse, trackball, or touchpad. Other input devices may include a microphone, a joystick, a game controller, a satellite dish, a scanner, a wireless device keyboard, voice commands, and the like. These and other input devices are typically connected to the processing unit 1120 through a user input 1140 and an associated interface coupled to the system bus 1121, but may also be connected through other interfaces and bus structures such as a parallel port, a game port, or a universal serial bus (USB). A graphics subsystem may also be connected to the system bus 1121. A projection unit in a projection display device, or a HUD in a viewing device or other type of display device may also be connected to the system bus 1121 via an interface such as an output interface 1150, which in turn may communicate with a video memory. In addition to a monitor, the computer may also include other peripheral output devices, such as speakers that may be connected through an output interface 1150.

[0140] The computer 1110 may operate in a networked or distributed environment using logical connections to one or more other remote computers, such as remote computers 1170, which in turn may have different media capabilities than the device 1110. The remote computer 1170 may be a personal computer, a server, a router, a network PC, a peer device, a personal digital assistant (PDA), a cell phone, a handheld computing device, a projection display device, a viewing device or other common network node, or any other remote media consumption or transmission device, and may include any or all of the elements described above with respect to the computer 1110. Figure 6The logical connections depicted in the figure include a network 1171, such as a local area network (LAN) or a wide area network (WAN), but may also include other networks / buses, either wired or wireless. Such a network environment is common in homes, offices, enterprise-wide computer networks, intranets, and the Internet.

[0141] When used in a LAN networking environment, the computer 1110 can be connected to the LAN 1171 through a network interface or adapter. When used in a WAN networking environment, the computer 1110 can typically include a communication component, such as a modem, or other means for establishing communications over the WAN, such as the Internet. Communication components such as wireless communication components, modems, etc., which can be internal or external, can be connected to the system bus 1121 via a user input interface of the input 1140 or other appropriate mechanism. In a networked environment, program modules or portions thereof described relative to the computer 1110 can be stored in a remote memory storage device. It should be understood that the network connections shown and described are exemplary, and other ways of establishing a communication link between computers can be used.

[0142] Example network environment

[0143] Figure 7 A schematic diagram of an exemplary networked or distributed computing environment 1200 is provided. The distributed computing environment includes computing objects 1210, 1212, etc. and computing objects or devices 1220, 1222, 1224, 1226, 1228, etc., which may include programs, methods, data storage, programmable logic, etc., as represented by applications 1230, 1232, 1234, 1236, 1238 and data storage device 1240. It is understood that computing objects 1210, 1212, etc. and computing objects or devices 1220, 1222, 1224, 1226, 1228, etc. may include different devices, including a multimedia display device or similar device depicted in the diagram, or other devices such as mobile phones, personal digital assistants (PDAs), audio / video devices, MP3 players, personal computers, laptops, etc. It should be further understood that data storage device 1240 may include one or more cache memories, one or more registers, or other similar data storage devices disclosed herein.

[0144] Each computing object 1210, 1212, etc. and computing objects or devices 1220, 1222, 1224, 1226, 1228, etc. may communicate directly or indirectly with one or more other computing objects 1210, 1212, etc. and computing objects or devices 1220, 1222, 1224, 1226, 1228, etc. via a communication network 1242. Figure 7 1242 is shown as a single element, but the communication network 1242 may also include Figure 7Other computing objects and computing devices that provide services for the system, and / or may represent multiple interconnected networks not shown. Each computing object 1210, 1212, etc. or computing objects or devices 1220, 1222, 1224, 1226, 1228, etc. may also include an application, such as application 1230, 1232, 1234, 1236, 1238, which may utilize an API or other object, software, firmware and / or hardware suitable for communicating with the techniques and disclosures described herein or implementing the techniques and disclosures described herein.

[0145] There are a variety of systems, components, and network configurations that support distributed computing environments. For example, computing systems can be connected together through wired or wireless systems, through local networks or widely distributed networks. Currently, many networks are coupled to the Internet, which provides an infrastructure for widely distributed computing and covers many different networks, but any network infrastructure can be used for exemplary communications, such as occur in system automatic diagnostic data collection as described in various embodiments herein.

[0146] Therefore, a large number of network topologies and network infrastructures can be utilized, such as client / server, peer-to-peer, or hybrid architectures. A "client" is a member of a class or group that uses the services of another class or group to which it is not related. A client can be a process, roughly a set of instructions or tasks, that requests a service provided by another program or process. The client process uses the requested service, in some cases without having to "know" any working details about the other program or service itself.

[0147] In a client / server architecture, especially a network system, a client is usually a computer that accesses shared network resources provided by another computer (such as a server). Figure 7 In the illustration, as a non-limiting example, computing objects or devices 1220, 1222, 1224, 1226, 1228, etc. can be considered clients and computing objects 1210, 1212, etc. can be considered servers, where computing objects 1210, 1212, etc. act as servers providing data services, such as receiving data from, storing data, processing data, and transmitting data to client computing objects or devices 1220, 1222, 1224, 1226, 1228, etc., although any computer can be considered a client, a server, or both, depending on the circumstances.

[0148] A server is typically a remote computer system accessible via a remote or local network, such as the Internet or a wireless network infrastructure. A client process may be active in a first computer system and a server process may be active in a second computer system, communicating with each other via a communication medium, thereby providing distributed functionality and allowing multiple clients to utilize the information gathering capabilities of the server. Any software object used in accordance with the techniques described herein may be provided independently or distributed across multiple computing devices or objects.

[0149] In a network environment where the communication network 1242 or bus is the Internet, for example, computing objects 1210, 1212, etc. may be web servers, and other computing objects or devices 1220, 1222, 1224, 1226, 1228, etc. may communicate with the web servers via any of a number of known protocols, such as the Hypertext Transfer Protocol (HTTP) or HTTPS. Computing objects 1210, 1212, etc. that act as servers may also act as clients, such as computing objects or devices 1220, 1222, 1224, 1226, 1228, etc., which may be a feature of a distributed computing environment.

[0150] References throughout the specification to "one embodiment," "an embodiment," "an example," "an implementation," "an aspect of the disclosure," or "an aspect" mean that a particular feature, structure, or characteristic described in conjunction with the embodiment, implementation, or aspect is included in at least one embodiment, implementation, or aspect of the disclosure. Thus, the appearances of the phrases "in one embodiment," "in an example," "in an aspect," "in an implementation," or "in an embodiment" in various places in this specification do not necessarily all refer to the same embodiment. Furthermore, particular features, structures, or characteristics may be combined in any suitable manner in the various disclosed embodiments.

[0151] As used herein, the terms "component," "system," "architecture," "engine," and the like are intended to refer to a computer or electronic related entity, hardware, a combination of hardware and software, software (e.g., in execution), or firmware. For example, a component may be one or more transistors, memory cells, an arrangement of transistors or memory cells, a gate array, a programmable gate array, an application specific integrated circuit, a controller, a processor, a process running on a processor, an object executable program or application that accesses or interfaces with semiconductor memory, a computer, etc., or a suitable combination thereof. The component may include erasable programming (e.g., processing instructions stored at least in part in an erasable memory) or hard programming (e.g., processing instructions burned into a non-erasable memory at the time of manufacture).

[0152] For example, processes executed from memory and processors can all be components. As another example, an architecture can include an arrangement of electronic hardware (e.g., parallel or serial transistors), processing instructions, and processors, which implement processing instructions in a manner suitable for the arrangement of electronic hardware. In addition, an architecture can include a single component (e.g., a transistor, a gate array, etc.) or an arrangement of components (e.g., a series or parallel arrangement of transistors, a gate array connected to a program circuit, a power lead, an electrical ground, an input signal line, and an output signal line, etc.). A system can include one or more components and one or more architectures. An example system can include a switch block architecture, which includes a cross input / output line and a transmission gate transistor, as well as a power supply, a signal generator, a communication bus, a controller, an I / O interface, an address register, etc. It should be understood that some overlaps in the definition are to be expected, and an architecture or system can be an independent component, or a component of another architecture, system, etc.

[0153] In addition to the above, the disclosed subject matter can be implemented as a method, device or product using typical manufacturing, programming or engineering techniques to produce hardware, firmware, software or any suitable combination thereof to control an electronic device to implement the disclosed subject matter. The terms "device" and "product" used herein are intended to cover electronic devices, semiconductor devices, computers or computer programs accessible from any computer-readable device, carrier or medium. Computer-readable media may include hardware media or software media. In addition, the medium may include non-transitory media or transmission media. In one example, non-transitory media may include computer-readable hardware media. Specific examples of computer-readable hardware media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips ...), optical disks (e.g., compact disks (CDs), digital versatile disks (DVDs) ...), smart cards and flash memory devices (e.g., cards, sticks, key drives ...). Computer-readable transmission media may include carrier waves, etc. Of course, those skilled in the art will recognize that many modifications may be made to the configuration without departing from the scope or spirit of the disclosed subject matter.

[0154] Unless otherwise indicated in the examples and elsewhere in the specification and claims, all parts and percentages are by weight, all temperatures are in degrees Celsius, and pressures are at or near atmospheric.

[0155] For any number or numerical range for a given property, a number or parameter within one range can be combined with another number or parameter from a different range for the same property to generate a numerical range.

[0156] Except in the operating examples, or where otherwise indicated, all numbers, values ​​and / or expressions referring to quantities of ingredients, reaction conditions, and the like used in the specification and claims are to be understood as being modified in all instances by the term "with respect to".

[0157] Although the present invention has been explained with respect to certain embodiments, it should be understood that various modifications thereof will be apparent to those skilled in the art after reading the specification. Therefore, it should be understood that the invention disclosed herein is intended to cover such modifications that fall within the scope of the appended claims.

Claims

1. A method for training a decoding model for decoding chord information from brain activity, wherein the decoding model is trained based on brain activity data of a subject while listening to music; The method comprises: Acquire music marked with chord tags, wherein the music marked with chord tags is music marked with chord information in advance; allowing a subject to listen to music labeled with chord labels and obtaining raw brain activity data from the subject by functional neuroimaging technology while the subject listens to the music labeled with chord labels; Extracting a brain activity pattern from the raw brain activity data, including encoding the three-dimensional raw brain activity data using spatial information and representing the properties of the brain activity pattern using the spatial pattern, encoding the two-dimensional raw brain activity data using channel information and representing the properties of the brain activity pattern using the spatiotemporal pattern; Establishing a correspondence between the extracted brain activity pattern and the chord label of the music listened to by the subject to form training data for the decoding model; The decoding model is trained based on the training data, the input of the decoding model is the brain activity pattern, the output is the chord information, and the trained decoding model is saved for use in decoding the chord information from the brain activity under the psychological music activity experience; The chord information includes a root note and / or a chord type, and the chord type carries information about emotional valence.

2. The method according to claim 1, wherein the functional neuroimaging technology includes one or more of functional magnetic resonance imaging, functional near-infrared spectroscopy, functional ultrasound imaging, non-invasive recording electroencephalogram, electrocorticography, intracortical recording, magnetoencephalography and positron emission tomography.

3. The method according to any one of claims 1 to 2, wherein: The architecture of the decoding model includes one or more of a dense neural network, a spatial convolutional neural network, a spatiotemporal convolutional neural network, a recurrent neural network, and a support vector machine.

4. A method for decoding chord information from brain activity, the steps comprising: 1) Load the decoding model that has been trained based on the brain activity data of the subjects while listening to music; 2) Prepare model input, including: 2a) Obtaining raw brain activity data from subjects via functional neuroimaging during psycho-music activities; 2b) extracting brain activity patterns from the raw brain activity data; 3) providing the extracted brain activity pattern as input to a decoding model that has been trained based on the brain activity data of the subject while listening to music to obtain the corresponding chord information; The decoding model is trained according to any one of claims 1-3.

5. The method of claim 4, further comprising fine-tuning the trained decoding model using unlabeled brain activity.

6. The method according to claim 4 or 5, wherein the functional neuroimaging uses one or more of the following techniques: functional magnetic resonance imaging, functional near-infrared spectroscopy, functional ultrasound imaging, non-invasive recording electroencephalography, electrocorticography, intracortical recording, magnetoencephalography and positron emission tomography.

7. The method according to claim 4, wherein: The original brain activity data of the subjects were obtained while they were listening to natural music.

8. The method according to claim 4, wherein: The raw brain activity data of the subjects were obtained while they were listening to the synthesized music.

9. The method according to claim 4, further comprising: The raw brain activity data is encoded using channel information and source reconstruction is performed.

10. The method according to claim 4, wherein: The psychological music activity includes one or more of music listening, music hallucination, music imagination and synesthesia.

11. A system for transcribing, generating and recording chords, comprising a learning module and a decoding module, wherein: The learning modules include: a functional neuroimaging component for measuring brain activity of a subject while listening to music tagged with chord labels, wherein the tagged chord music is music pre-tagged with chord information; a signal processing component for extracting brain activity patterns, wherein the signal processing component uses spatial information to encode three-dimensional raw brain activity data and uses spatial patterns to represent properties of brain activity patterns, and uses channel information to encode two-dimensional raw brain activity data and uses spatiotemporal patterns to represent properties of brain activity patterns, a well-defined database of chord labels for the music listened to by the subject, and A decoding model with a predefined architecture for training; and The decoding module comprises: Functional neuroimaging components for measuring primary brain activity during various psychomusical activities, signal processing components to extract brain activity patterns appropriate to the input, a trained decoding model derived from the learning module for converting input data into chord information, and a data output component configured to output chord information from the trained decoding model, wherein the chord information includes a root note and / or a chord type, the chord type carrying information about emotional valence; The decoding model is trained according to any one of claims 1-3.

12. The system according to claim 11, wherein: The functional neuroimaging component uses one or more of the following techniques to perform measurements: functional magnetic resonance imaging, functional near-infrared spectroscopy, functional ultrasound imaging, non-invasively recorded electroencephalogram, electrocorticography, intracortical recording, magnetoencephalography, and positron emission tomography.

13. The system according to claim 11, wherein: The architecture of the decoding model includes one or more of a dense neural network, a spatial convolutional neural network, a spatiotemporal convolutional neural network, a recurrent neural network, and a support vector machine.

14. A system for a chord decoding protocol, comprising: A memory storing a functional unit and a processor executing the functional unit stored in the memory, wherein the functional unit comprises: A neural code extraction model that generates raw data from existing music-related neuroimaging datasets and offline neural measurements of users while listening to music, and then extracts processed brain activity patterns, i.e., chord-related neural codes, from the raw data; A decoding model, whose function is to learn and estimate an approximate mapping from the neural code to internal musical chords; and The trained model converts the neural code in the application scenario into an estimate of the chord information and can perform fine-tuning operations; The decoding model is trained according to any one of claims 1-3.

15. The system of claim 14, wherein: The architecture of the decoding model includes one or more of a dense neural network, a spatial convolutional neural network, a spatiotemporal convolutional neural network, a recurrent neural network, and a support vector machine.

Images