Generative music closed-loop regulation and control method and system based on reinforcement learning model
Through the music closed-loop regulation system based on the reinforcement learning model, the problem of lack of real-time feedback and adjustment of music therapy is solved by using brain wave signals, and the dynamic adjustment of personalized music playback strategies is achieved, and the effectiveness of music therapy is improved.
Patent Information
- Application Number
- CN202411167537.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2025-04-18
AI Technical Summary
Existing music therapy lacks real-time feedback and adjustments to individual physiological status, resulting in limited efficacy.
A generative music closed-loop regulation method and system based on reinforcement learning model is adopted to conduct personalized music closed-loop regulation based on the user's physiological and psychological state through real-time acquisition and analysis of brain wave signals.
It realizes dynamic adjustment of music playback strategies based on users' real-time EEG feedback to improve the effectiveness and personalization of music therapy.
Smart Images
Figure CN119848415A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of wireless audio equipment for brain wave signals, and in particular to a generative music closed-loop control method and system based on a reinforcement learning model. Background Art
[0002] In today's fast-paced life, more and more people are facing problems such as sleep disorders, excessive stress, and decreased concentration. Traditional solutions include medication and music therapy. Although medication is effective, it may be accompanied by side effects and the risk of addiction. Music therapy is widely welcomed for its non-invasive and side-effect-free characteristics. However, existing music therapies are usually based on fixed tracks and lack real-time feedback and adjustment of individual physiological states, resulting in limited efficacy.
[0003] Electroencephalogram (EEG) technology provides a means to monitor brain activity in real time. By analyzing EEG signals, we can obtain information such as the user's emotional state, cognitive load, and sleepiness level. Combining EEG technology with closed-loop music regulation can perform personalized closed-loop music regulation based on the user's current physiological and psychological state, thereby improving the effectiveness of music therapy. However, the technical problems of combining existing EEG technology with closed-loop music regulation mainly involve real-time acquisition and precise analysis of brain wave signals, personalized closed-loop music regulation and real-time adaptation, effective application and optimization of reinforcement learning algorithms, and design and implementation of closed-loop feedback mechanisms. Summary of the invention
[0004] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a generative music closed-loop control method and system based on a reinforcement learning model, which can solve the existing problems.
[0005] In order to achieve the above object, the technical solution of the present invention is as follows:
[0006] The present invention is implemented by the following technical solution: a generative music closed-loop control method based on a reinforcement learning model, comprising the following steps:
[0007] Step 1, initializing parameters, including initializing action parameters N=0, maximum number of actions Nm, and minimum data volume M for decision module to start training;
[0008] Step 2: Use the brain wave collection module to collect the user's brain wave information within a period of time;
[0009] Step 3: Process the acquired brain wave information using a data processing module;
[0010] Step 4: Store the processed brain wave information into the database module and determine whether the amount of data in the database > M is established.
[0011] If it is established, train and update the decision module according to the method of training the model in reinforcement learning, and then proceed to step 5;
[0012] If not, go directly to step 5;
[0013] Step 5: The decision-making module makes closed-loop control decisions based on the data processed by the data processing module;
[0014] Step 6: The music closed-loop control module controls the corresponding music playing strategy according to the closed-loop control decision;
[0015] Step 7: The music playing module plays music according to the music playing strategy;
[0016] Step 8: Collect the user's brain wave information within a period of time again, and the user state acquisition module determines whether the user has reached the target state, and sets N=N+1;
[0017] If so, the task ends;
[0018] If not, determine whether N is equal to Nm;
[0019] If so, the task ends;
[0020] If not, proceed to step three.
[0021] Furthermore, the brain wave collection module uses head-mounted electrodes or ear-mounted electrodes to collect the user's brain wave information.
[0022] Furthermore, the acquired brain wave information is processed by the data processing module, including filtering, outlier deletion and feature extraction.
[0023] Furthermore, the music playing strategy includes music type, volume and playing speed.
[0024] Furthermore, the decision module includes a machine learning model formed through training, and the decision module outputs the token required by the music closed-loop control module based on the data closed-loop control decision processed by the data processing module; the music closed-loop control module includes a closed-loop control model formed through training, and the music closed-loop control module receives the token and performs closed-loop control of the music playback strategy.
[0025] Furthermore, the user status acquisition module determines whether the user has reached the target status, including:
[0026] The user status acquisition module obtains the EEG information collected by the EEG acquisition module within a period of time;
[0027] Acquire status information related to the task status based on EEG information;
[0028] Determine whether the target state has been reached based on relevant state information.
[0029] Furthermore, the target state includes a sleeping state, a concentration state or a relaxation state.
[0030] A generative music closed-loop control system based on a reinforcement learning model, comprising:
[0031] A data collection module, which is used to collect brain wave information of the user over a period of time;
[0032] A data processing module, the data processing module is connected to the data acquisition module, and the data processing module is used to process the acquired brain wave information;
[0033] A decision-making module, the decision-making module is connected to the data processing module, and the decision-making module is used to make closed-loop control decisions based on the data processed by the data processing module;
[0034] A music closed-loop control module, the music closed-loop control module is connected to the decision module, and the music closed-loop control module controls the corresponding music playing strategy according to the decision of the closed-loop control;
[0035] A music playing module, the music playing module is connected to the music closed-loop control module, and the music playing module plays music according to the music playing strategy;
[0036] A user status acquisition module, the user status acquisition module is connected to the data acquisition module, and the user status acquisition module is used to determine whether the user has reached the target state;
[0037] A database module, wherein the database module is connected to the data processing module and the decision-making module, and the database module is used for reinforcement learning training and updating the decision-making module.
[0038] A computer-readable storage medium stores a computer program, wherein the computer program stores instructions, and when the instructions are executed by one or more processors, the one or more processors perform any of the methods described above.
[0039] The present invention provides a generative music closed-loop control method and system based on a reinforcement learning model, which can design different control technologies for different scenarios, so that users can reach a certain target state more efficiently and quickly, and can be applied in the fields of sleep aid, improving attention, music relaxation, etc. The present invention collects brain wave information and personalizes the closed-loop control of music according to the user's brain wave state. Compared with the user's own selection of music control, it is more scientific and efficient. Compared with consulting a professional music control technician, it is easier to obtain a customized music control strategy, which is convenient and convenient; at the same time, according to the user's brain wave state, the user's current brain wave characteristics can be extracted to provide a highly targeted music control strategy, which is highly personalized. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0041] Figure 1 This is a flow chart of a generative music closed-loop control method based on a reinforcement learning model according to an implementation example of the present invention. DETAILED DESCRIPTION
[0042] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0043] The present invention provides a generative music closed-loop control method based on a reinforcement learning model, such as Figure 1 As shown, the following steps are included:
[0044] Step 1, initializing parameters, including initializing action parameters N=0, maximum number of actions Nm, and minimum data volume M for decision module to start training;
[0045] Step 2: using a brain wave acquisition module to collect brain wave information of the user within a period of time; specifically, the brain wave acquisition module uses head-mounted electrodes or ear-mounted electrodes to collect brain wave information of the user;
[0046] Step 3: Use the data processing module to process the acquired brain wave information; specifically, it includes filtering, outlier removal and feature extraction processing; for example, EEG signals usually contain multiple frequency components, but brain activity is mainly concentrated between 0.5Hz and 100Hz. The purpose of filtering is to remove unnecessary noise and retain the signal within the required frequency range. The following filtering techniques can be used for filtering, including but not limited to:
[0047] Bandpass filtering: Apply a bandpass filter (such as Butterworth, Chebyshev, or elliptic), usually with a lower limit of 0.5 Hz and an upper limit of 70 Hz or higher, depending on the purpose of the study.
[0048] Notch filtering: used to remove interference of specific frequencies, such as 50Hz or 60Hz AC power noise, and their harmonics.
[0049] EEG data may contain outliers caused by muscle movement, blinking, or poor device contact. Identifying and removing these outliers is critical to maintaining data quality; outlier removal methods include but are not limited to:
[0050] Visual inspection: Manual inspection of raw data records to mark and remove obvious artifacts.
[0051] Statistical methods: Calculate the mean and standard deviation of each channel data, identify data points outside the normal range, and correct or delete them.
[0052] Independent Component Analysis (ICA): ICA is a commonly used technique for isolating independent sources in a signal and can identify and exclude those components that represent noise.
[0053] Feature extraction is the process of extracting meaningful information from the raw EEG signal. These features can be used for subsequent classification or analysis tasks. Including but not limited to: Frequency domain features: such as power spectral density (PSD), which can be obtained by Fourier transform, which reveals the energy distribution of the signal at different frequencies. Time domain features: such as mean, variance, kurtosis, skewness, and various time domain statistical features. Time-frequency features: including wavelet transform and short-time Fourier transform (STFT), which are used to analyze the time-varying spectral characteristics of the signal. Connectivity features: such as phase locking value (PLV), coherence and Granger causality, which are used to evaluate the interaction between different brain regions.
[0054] Step 4: Store the processed brain wave information into the database module. Specifically, the brain wave data used to obtain the token, the token, the reward, and the brain wave data of the user after the music playback module is executed are used for reinforcement learning training and updating the decision model; determine whether the amount of data in the database > M is established.
[0055] If it is established, train and update the decision module according to the method of training the model in reinforcement learning, and then proceed to step 5;
[0056] If not, go directly to step 5;
[0057] The decision module is trained and updated according to the method of training the model in reinforcement learning, mainly by updating the decision module through reinforcement learning, including extracting meaningful features from the processed EEG information, including but not limited to power spectral density, frequency band energy, connectivity, which may be related to music preference or emotional state. Label or reward definition: Determine which music clips are associated with positive EEG patterns based on user feedback or some objective indicators (such as heart rate changes, skin conductance response, etc.), and use this as a reward signal.
[0058] Designing a reinforcement learning model mainly involves defining states: states may include the type of music currently being played, EEG feature vectors, etc. Then defining actions: actions can be switching to a different type of music, adjusting the volume, changing the playback speed, etc. Designing a reward function: the reward should reflect the degree of impact of music playback on the user's brain waves, and encourage actions that lead to positive EEG responses.
[0059] Model training, initialization: Set the initial policy and value function. Interaction and learning: The agent performs actions while the user listens to music, collects EEG data, and updates the policy and value function based on the reward function. This may involve value-based learning (such as Q-learning), policy-based learning (such as REINFORCE), or Actor-Critic methods.
[0060] Dynamic decision updates: Over time, the model dynamically adjusts the music playback strategy based on the user's real-time EEG feedback. For example, if the model detects that the user is in a relaxed state, it may increase the proportion of soft music; if the user appears anxious, it may play more cheerful melodies.
[0061] Step 5: The decision module makes closed-loop control decisions based on the data processed by the data processing module, and outputs the token required by the music closed-loop control module. The decision module is to control a machine learning model through closed-loop training, which mainly includes: 1. First, clarify the goal and define the problem; 2. Collect the brain wave data and status of the crowd; 3. Determine the machine learning model and train it based on the collected data; 4. Evaluate and optimize the model to obtain the final decision model.
[0062] Step six, the music closed-loop control module controls the corresponding music playback strategy according to the closed-loop control decision; specifically, the music closed-loop control module receives the token and controls the music playback strategy in a closed-loop manner; the music playback strategy includes the type of music, volume, and playback speed. The music closed-loop control module is a closed-loop control model based on training closed-loop control; it mainly includes: 1. Data collection; 2. Design of model architecture; 3. Model initialization; 4. Encoding input; 5. Loss function and optimizer; 6. Training loop; 7. Regularization and avoiding overfitting; 8. Evaluation and debugging to obtain the final music closed-loop control module;
[0063] Step 7: The music playing module plays music according to the music playing strategy;
[0064] Step 8: Collect the user's brain wave information within a period of time again, and the user state acquisition module determines whether the user has reached the target state, and sets N=N+1;
[0065] If so, the task ends;
[0066] If not, determine whether N is equal to Nm;
[0067] If so, the task ends;
[0068] If not, proceed to step three.
[0069] The user status acquisition module determines whether the user has reached the target status, including:
[0070] The user status acquisition module obtains the EEG information collected by the EEG acquisition module within a period of time;
[0071] Acquire status information related to the task status based on EEG information;
[0072] Determine whether the target state has been reached based on relevant state information.
[0073] The target state includes a sleeping state, a concentration state, or a relaxation state.
[0074] The present invention also provides a generative music closed-loop control system based on a reinforcement learning model, comprising:
[0075] A data collection module, which is used to collect brain wave information of the user over a period of time;
[0076] A data processing module, the data processing module is connected to the data acquisition module, and the data processing module is used to process the acquired brain wave information;
[0077] A decision module, the decision module is connected to the data processing module, and the decision module is used to make closed-loop control decisions based on the data processed by the data processing module; illustratively, the decision module includes a machine learning model, the input is the processed brain wave information data, and the output is a token based on the closed-loop control of the user's current brain wave;
[0078] A music closed-loop control module, the music closed-loop control module is connected to the decision module, and the music closed-loop control module controls the corresponding music playing strategy according to the decision of the closed-loop control; exemplarily, the music closed-loop control module includes a closed-loop control model, which can close-loop control the music according to the token of the closed-loop control in the decision module, so as to achieve the effect of regulating the user's brain waves. The music closed-loop control module is pre-trained and frozen in all processes, and the music library used for training is a music library designed specifically for the task, such as a hypnotic music library or a focus music library.
[0079] A music playing module, the music playing module is connected to the music closed-loop control module, and the music playing module plays music according to the music playing strategy;
[0080] A user status acquisition module, the user status acquisition module is connected to the data acquisition module, and the user status acquisition module is used to determine whether the user has reached the target state;
[0081] A database module, wherein the database module is connected to the data processing module and the decision-making module, and the database module is used to strengthen learning training and update the decision-making module; specifically, the database module is used to obtain brain wave data of the token, the token, the reward, and the brain wave data of the user after the music playback module is executed, which is used to strengthen learning training and update the model in the user state acquisition module.
[0082] Implementation case: The present invention is applied to a sleep-aiding scenario, the target state is a sleep state, and the specific operation steps are as follows:
[0083] 1. Use sleep-aiding music to train the sleep-aiding music closed-loop control model;
[0084] 2. Train the decision model in the training population according to the training process;
[0085] 3. Initialize the number of actions N = 0, the maximum number of actions Nm;
[0086] 4. The brain wave acquisition module collects brain waves for a processing length of time;
[0087] 5. The data processing module processes the EEG data, including filtering, outlier deletion, and extraction of feature values that can characterize the user's sleepiness;
[0088] 6. The decision-making module makes decisions based on the data processed by the data processing module and outputs the token required by the closed-loop control module;
[0089] 7. The music closed-loop control module receives the token and performs closed-loop control of the sleep-aiding music, including specific sleep-aiding music and volume information;
[0090] 8. The music playback module plays closed-loop controlled sleep-aiding music;
[0091] 9. Collect brain waves for another processing period of time to determine whether the user has fallen asleep, and N becomes N+1;
[0092] If the user has fallen asleep or N = the maximum number of actions Nm, the task ends;
[0093] If the above two states are not reached, return to 5 and transfer the data back to 5;
[0094] The present invention also protects a computer-readable storage medium having a computer program stored thereon, wherein the computer program stores instructions which, when executed by one or more processors, cause the one or more processors to perform any of the methods described above.
[0095] The present invention designs different control technologies for different scenarios, allowing users to reach a certain target state more efficiently and quickly, and can be applied in the fields of sleep aid, improving attention, music relaxation, etc. It has the ability to personalize closed-loop control of music according to the user's EEG state, which is more scientific and efficient than users selecting music control by themselves. It is convenient and fast, and it is easier to obtain customized music control strategies than consulting professional music control technicians. It is highly personalized and can extract the user's current EEG characteristics according to the user's EEG state, providing them with highly targeted music control strategies.
[0096] Those of ordinary skill in the art will appreciate that the unit-level algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this article.
[0097] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0098] In the several embodiments provided herein, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.
[0099] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this article.
[0100] In addition, each functional unit in each embodiment of this invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional unit.
[0101] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this article is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this article. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.
[0102] Specific embodiments are used in this article to illustrate the principles and implementation methods of this article. The description of the above embodiments is only used to help understand the methods and core ideas of this article. At the same time, for general technicians in this field, according to the ideas of this article, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on this article.
Claims
1. A generative music closed-loop control method based on a reinforcement learning model, characterized in that: The following steps are involved: Step 1, initializing parameters, including initializing action parameters N=0, maximum number of actions Nm, and minimum data volume M for decision module to start training; Step 2: Use the brain wave collection module to collect the user's brain wave information within a period of time; Step 3: Process the acquired brain wave information using a data processing module; Step 4: Store the processed brain wave information into the database module and determine whether the amount of data in the database > M is established. If it is established, train and update the decision module according to the method of training the model in reinforcement learning, and then proceed to step 5; If not, go directly to step 5; Step 5: The decision-making module makes closed-loop control decisions based on the data processed by the data processing module; Step 6: The music closed-loop control module controls the corresponding music playing strategy according to the closed-loop control decision; Step 7: The music playing module plays music according to the music playing strategy; Step 8: Collect the user's brain wave information within a period of time again, and the user state acquisition module determines whether the user has reached the target state, and sets N=N+1; If so, the task ends; If not, determine whether N is equal to Nm; If so, the task ends; If not, proceed to step three.
2. A generative music closed-loop control method based on a reinforcement learning model according to claim 1, characterized in that: The brain wave acquisition module uses head-mounted electrodes or ear-mounted electrodes to acquire the user's brain wave information.
3. A generative music closed-loop control method based on a reinforcement learning model according to claim 1, characterized in that: The data processing module is used to process the acquired brain wave information; Including filtering, outlier removal and feature extraction processing.
4. A generative music closed-loop control method based on a reinforcement learning model according to claim 1, characterized in that: The music playing strategy includes music type, volume and playing speed.
5. The generative music closed-loop control method based on reinforcement learning model according to claim 1, characterized in that: The decision module includes a machine learning model formed through training. The decision module makes a closed-loop control decision based on the data processed by the data processing module, and outputs the token required by the music closed-loop control module; the music closed-loop control module includes a closed-loop control model formed through training. The music closed-loop control module receives the token and performs closed-loop control of the music playback strategy.
6. A generative music closed-loop control method based on a reinforcement learning model according to claim 1, characterized in that: The user status acquisition module determines whether the user has reached the target status, including: The user status acquisition module obtains the EEG information collected by the EEG acquisition module within a period of time; Acquire status information related to the task status based on EEG information; Determine whether the target state has been reached based on relevant state information.
7. A generative music closed-loop control method based on a reinforcement learning model according to claim 6, characterized in that: The target state includes a sleeping state, a concentration state, or a relaxation state.
8. A generative music closed-loop control system based on a reinforcement learning model, characterized in that: include: A data collection module, which is used to collect brain wave information of the user over a period of time; A data processing module, the data processing module is connected to the data acquisition module, and the data processing module is used to process the acquired brain wave information; A decision-making module, the decision-making module is connected to the data processing module, and the decision-making module is used to make closed-loop control decisions based on the data processed by the data processing module; A music closed-loop control module, the music closed-loop control module is connected to the decision module, and the music closed-loop control module controls the corresponding music playing strategy according to the decision of the closed-loop control; A music playing module, the music playing module is connected to the music closed-loop control module, and the music playing module plays music according to the music playing strategy; A user status acquisition module, the user status acquisition module is connected to the data acquisition module, and the user status acquisition module is used to determine whether the user has reached the target state; A database module, wherein the database module is connected to the data processing module and the decision-making module, and the database module is used for reinforcement learning training and updating the decision-making module.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program stores instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 7.
Citation Information
Cited By
Situation interactive foreign language training system based on virtual reality
CN120544439A