Musical instrument playing skill optimization method based on biological signal feedback

Abstract

The invention discloses a musical instrument playing skill optimization method based on biological signal feedback. The method comprises the following steps: S1, collecting a multi-modal biological signal; s2, generating a physiological signal feature vector through a deep convolutional network; s3, generating a physiological state vector of the player by using the multi-mode self-attention network; s4, acquiring micro-action data of the player in real time; s5, establishing an action error model through a support vector machine, and calculating an error correction amount; s6, using a simulated annealing algorithm to optimize the micro-action of the player, and dynamically adjusting playing action parameters; s7, collecting a playing audio in real time, analyzing an audio signal by using fast Fourier transform, and dynamically optimizing the tone quality in combination with the physiological state vector of the player; and S8, carrying out collaborative modeling on the biological signal and the audio signal through a graph neural network and a Bayesian optimization algorithm, and adjusting the action mode and the tone quality of the player in real time. According to the invention, through collaborative modeling of biological signal feedback and audio signals, real-time musical instrument playing skill optimization is realized.

Description

Technical Field

[0001] The present invention relates to the field of biological signal processing and artificial intelligence technology, and in particular to a method for optimizing musical instrument playing skills based on biological signal feedback. Background Art

[0002] With the development of science and technology, biosignal feedback is increasingly used in various fields, especially in health monitoring, sports training and cognitive training, and has achieved remarkable results. In the field of musical instrument performance, how to monitor and optimize the performer's movements in real time through biosignal feedback has become an important direction for improving musical instrument learning and performance techniques. Traditional methods for optimizing musical instrument performance skills mainly rely on the guidance of teachers and the self-feedback of performers, lacking a real-time, biosignal-driven optimization mechanism, and still have great limitations, especially in fine movement adjustment and sound quality optimization.

[0003] Existing instrument playing technique optimization technologies usually evaluate the skill level of the performer by observing the performer's playing state, using some traditional means, such as verbal guidance from teachers, recording playback analysis, etc. Traditional methods often rely on the performer's self-perception or the teacher's intuitive feedback, which is inefficient and lacks precision. During training, performers cannot obtain specific feedback on movement errors in a timely manner, resulting in slow progress in improving playing skills. Although some advanced motion capture systems and motion analysis software can assist in evaluating the performer's movements, these technologies mainly focus on the capture and analysis of macroscopic movements, and do not provide sufficiently detailed analysis of tiny playing movements (such as fine-tuning of fingers, slight rotation of wrists, etc.), and lack comprehensive consideration of physiological signals.

[0004] At present, biosignal feedback technology has been applied in some fields such as sports training and health monitoring, but its application in optimizing musical instrument playing skills is still relatively preliminary. Biosignals, such as electromyography (EMG), electroencephalography (EEG), heart rate (HR), etc., can reflect the performer's physiological state, emotions, and degree of concentration. These signals are closely related to playing skills. By analyzing these signals, performers can understand their own physiological state and adjust their playing movements. However, existing technologies make it difficult to effectively combine these biosignals with playing skills, and most methods lack the ability to capture and optimize feedback on the performer's micro-movements, resulting in the failure of the application of biosignals to fully realize their potential in improving playing skills.

[0005] In terms of sound quality optimization, existing technologies mainly rely on the performer's ears and intuitive perception for adjustments, and lack systematic optimization based on scientific principles. For example, although evaluating sound quality by recording playback can provide a certain reference, the subjectivity of sound quality makes this method difficult to objectively and accurately quantify and optimize. In addition, existing sound quality optimization technologies fail to combine the performer's physiological signals and lack a sound quality adjustment mechanism for individual differences among performers. The performer's physiological state, especially factors such as muscle fatigue and attention level, directly affects the stability of their performance and the performance of sound quality, but existing technologies cannot dynamically optimize the sound quality according to the performer's physiological state in real time.

[0006] In addition, existing motion optimization algorithms, such as traditional motion analysis and error correction techniques, usually rely on static or regularized optimization methods, which are difficult to cope with the complex dynamic changes during the performance. Existing motion correction technologies often seem powerless when faced with the performer's micro-motion adjustments and cannot achieve refined motion optimization. Although some efficient optimization algorithms, such as simulated annealing and genetic algorithms, have achieved certain results in other fields, there is still a lack of effective methods to combine these algorithms with the performer's biological signals in the optimization of musical instrument playing skills.

[0007] Therefore, how to provide a method for optimizing musical instrument playing skills based on biosignal feedback is an urgent problem that needs to be solved by those skilled in the art. Summary of the invention

[0008] One purpose of the present invention is to propose a method for optimizing musical instrument playing skills based on biosignal feedback. The present invention collects the performer's biosignals and audio signals, combines deep learning and optimization algorithms, and adjusts the performer's action patterns and sound quality characteristics in real time. The simulated annealing algorithm, graph neural network and Bayesian optimization algorithm are used to achieve dynamic optimization of playing skills. This method not only improves the performer's technical performance, but also can make precise adjustments based on physiological state and audio feedback. It has the advantages of efficient adaptability, precise optimization effect and real-time feedback adjustment, which significantly improves the quality of musical instrument performance and the performer's performance ability.

[0009] A method for optimizing musical instrument playing skills based on biosignal feedback according to an embodiment of the present invention comprises the following steps:

[0010] S1, collecting multimodal bio-signals of the performer through a sensor array;

[0011] S2, extracting features from the collected multimodal biological signals, identifying physiological features associated with playing skills through a deep convolutional neural network, and generating physiological signal feature vectors;

[0012] S3, based on the generated physiological signal feature vector, using the multimodal self-attention network to perform weighted fusion of various biological signals to generate the performer's physiological state vector;

[0013] S4, using flexible piezoelectric sensors, accelerometers and gyroscopes to collect the player's micro-motion data in real time, monitoring fingertip pressure, wrist rotation angle and finger sliding on the keys;

[0014] S5, based on the collected micro-motion data, establish a motion error model through a support vector machine, analyze the performer's motion error, and calculate the error correction amount;

[0015] S6, optimizing the player's micro-movements using a simulated annealing algorithm according to the error correction amount, and dynamically adjusting the performance action parameters through global optimization, wherein the performance action parameters include fingering, grip strength, and wrist angle;

[0016] S7, through the adjusted performance action parameters, real-time collection of performance audio, and analysis of audio signals using fast Fourier transform, and dynamic optimization of sound quality in combination with the performer's physiological state vector;

[0017] S8. Use graph neural networks and Bayesian optimization algorithms to collaboratively model biological signals and audio signals, establish an optimization closed loop, and adjust the performer's movement pattern and sound quality in real time.

[0018] Optionally, the multimodal bio-signal specifically includes electromyography, electroencephalography, heart rate and galvanic skin response, which are used to reflect muscle fatigue, neural response, emotional fluctuations and concentration during the performance.

[0019] Optionally, the S3 specifically includes:

[0020] S31, receiving the generated physiological signal feature vector and converting it into a unified format, wherein the physiological signal feature vector includes multimodal biological signal data from the performer, wherein each signal type represents a different physiological feature of the performer during the performance;

[0021] S32. Use the multimodal self-attention network to perform multi-angle fusion calculations on different signal modalities, perform linear transformation on the physiological signal feature vector, obtain the query vector, key vector and value vector, and introduce a multi-head attention mechanism to calculate different weighted values ​​of multiple signal modalities:

[0022]

[0023] Among them, α i,h represents the hth head attention weight of the i-th biological signal, exp represents the exponential function, N represents the total category of biological signals, and d h represents the dimension of the h-th attention head, Q irepresents the query vector of the i-th biological signal, K i represents the key vector of the i-th biological signal, K j represents the key vector of the j-th biological signal;

[0024] S33, using the calculated attention weight α i,h The characteristic vectors of various physiological signals are weighted to generate weighted signal characteristic vectors, and a position encoding mechanism is introduced to process the positional relationship between different signals:

[0025]

[0026] Among them, V fused represents the weighted physiological signal feature vector, H represents the number of attention heads, and X i represents the feature vector of the i-th biological signal, P i Represents the positional encoding of the i-th type of biological signal;

[0027] S34, by weighting the physiological signal feature vector V fused Perform nonlinear transformation and apply deep neural network to perform high-order feature mapping on the signal:

[0028] V transformed =σ(W f ·V fused +b f );

[0029] Among them, V transformed represents the transformed physiological signal feature vector, σ represents the activation function, W f represents the weight matrix, b f represents the bias term;

[0030] S35, the transformed physiological signal feature vector V transformed Combined with the performer's current performance action parameter vector, the association between physiological signals and action parameters is strengthened through a joint embedding model:

[0031] V final =V transformed ⊕A;

[0032] Among them, ⊕ represents the vector concatenation operation, V final represents the final fused physiological state vector, and A represents the performer's current performance action parameter vector;

[0033] S36, through the last layer of the fully connected network, the final fused physiological state vector V final Further optimization and adjustment are performed to generate the performer's physiological state vector.

[0034] Optionally, the S5 specifically includes:

[0035] S51, extract features from the collected micro-motion data to obtain the feature vector F of each action features =[f1,f2,…,f m ], where each feature f i Represents real-time data of the performer’s micro-movements during performance;

[0036] S52. Use feature selection methods to select a feature set with high discrimination from micro-motion data And build a training set in represents the feature vector of the jth sample, M represents the total number of samples, y (j) represents the action error value corresponding to the jth sample, reflecting the difference between the performer's current action and the ideal action;

[0037] S53. Use support vector machine to train the training set T to obtain the optimal classification hyperplane. The optimization goal of support vector machine is to maximize the interval, and the support vector is calculated by Lagrange multiplier method:

[0038]

[0039] Among them, f(F selected ) represents the decision function, δ i represents the Lagrange multiplier, represents the feature vector of the i-th sample, b represents the bias term, y (i) Represents the action error value corresponding to the i-th sample;

[0040] S54, use the trained support vector machine to predict the new micro-motion data, input the newly collected motion features into the support vector machine, and output the current motion error value E i ;

[0041] S55, based on the current action error value E i Constructing the weighted error model E weighted , introduce the weight coefficient ω of each action error i Calculate the weighted error value:

[0042]

[0043] Among them, E weighted represents the weighted error value, represents the variance of the i-th error value, and m represents the total number of error values;

[0044] S56, based on the weighted error value E weighted Calculate the error correction:

[0045] ΔD=K·E weighted +λ·∥E weighted ∥2;

[0046] Where ΔD represents the error correction, K represents the adjustment matrix, λ represents the regularization coefficient, ∥E weighted ∥2 represents the L2 norm of the weighted error.

[0047] Optionally, the S6 specifically includes:

[0048] S61, setting initial conditions of the simulated annealing algorithm, wherein the initial conditions include an initial temperature and a temperature drop rate, and setting an initial solution of the performance action parameters to be initial micro-action data of the performer, including fingering, grip strength, and wrist angle;

[0049] S62, generating a new candidate solution according to the error correction amount, wherein the candidate solution is generated by perturbing the current performance action parameters, and the perturbation amplitude is controlled by the current temperature; the perturbation process includes fine-tuning the fingering, grip strength and wrist angle, simulating the optimization action that may occur during the performance of the performer, and the generated candidate solution is represented as a set of new performance action parameters;

[0050] S63, calculating the error between the generated candidate solution and the target action, wherein the error represents the optimization effect of the candidate solution, a smaller error represents that the candidate solution is closer to the target action, and a larger error represents that the candidate solution has a larger room for improvement;

[0051] S64, if the error of the candidate solution is less than the error of the current solution, the candidate solution is accepted as the new current solution; if the error of the candidate solution is greater than the error of the current solution, the candidate solution is accepted according to a certain probability, and the certain probability is affected by the current temperature, and the higher the temperature, the greater the acceptance probability;

[0052] S65. After each round of iteration, the simulated annealing algorithm gradually reduces the temperature according to the set cooling rate. The temperature drop means that the amplitude of the disturbance decreases, and the search process gradually concentrates near the local optimal solution. As the number of iterations increases, the optimization process gradually converges until a solution close to the optimal solution is found.

[0053] S66: When the temperature drops to a preset minimum value or the simulated annealing algorithm iterations reach a maximum number, the simulated annealing algorithm is stopped, and the optimization result is the final performance action parameter.

[0054] Optionally, the S7 specifically includes:

[0055] S71, collecting the performer's audio signal in real time through a microphone array, wherein the duration of the audio signal collection is synchronized with the performance process;

[0056] S72, converting the collected audio signal into a frequency domain signal by fast Fourier transform, obtaining spectrum data in each time period, and extracting amplitude information of the audio signal at different frequencies by fast Fourier transform;

[0057] S73, extracting audio spectrum features of the audio signal based on the spectrum data, the audio spectrum features including spectrum amplitude, fundamental frequency, harmonic components and bandwidth; the spectrum amplitude reflects the intensity change of the audio signal, and the fundamental frequency and harmonic components help to describe the pitch and timbre characteristics of the audio;

[0058] S74, combining the audio spectrum feature with the player's physiological state vector, and performing joint processing through a weighted fusion algorithm, so that each frequency component in the audio signal is associated with the corresponding physiological state;

[0059] S75. Based on the comprehensive audio spectrum characteristics and the performer's physiological state vector, the audio signal is dynamically optimized by a sound quality optimization algorithm, wherein the sound quality optimization algorithm dynamically adjusts the performer's action mode and sound quality characteristics based on the sound quality adjustment parameters fed back in real time:

[0060]

[0061] Among them, Q opt represents the optimized sound quality, p represents the adjusted performance action parameter vector, λ1 and λ2 represent weight coefficients, F(X,p) represents the relationship function between the audio signal spectrum characteristics and the action parameters under the current performance action parameters, T represents the target sound quality characteristics, X represents the spectrum feature vector extracted from the audio signal, G(S,p) represents the relationship function between the physiological state vector and the performance action parameters, S ref represents the ideal physiological state vector, and S represents the performer's physiological state vector.

[0062] Optionally, the S8 specifically includes:

[0063] S81. Construct a graph neural network, where the biological signals and audio signals are used as nodes in the graph, and the edges between the nodes represent the relationship between the signals.

[0064] S82, generating a final representation vector of each node by aggregating information of adjacent nodes layer by layer in the graph neural network, wherein the final representation vector includes comprehensive features of the biological signal and the audio signal;

[0065] S83, applying a Bayesian optimization algorithm to optimize the node representation output by the graph neural network, wherein the Bayesian optimization algorithm adjusts the parameters of the graph neural network through Gaussian process regression so that the mapping relationship between the biological signal and the audio signal is optimized;

[0066] S84, dynamically adjusting the performer's action pattern and sound quality parameters under the guidance of the Bayesian optimization algorithm, the Bayesian optimization algorithm derives optimized action parameters by evaluating each possible action pattern;

[0067] S85. Based on the optimization results of the graph neural network and the Bayesian optimization algorithm, the performer's movement pattern and sound quality parameters are adjusted through real-time feedback to form a closed-loop control mechanism; the closed-loop control mechanism dynamically updates the performer's movement pattern according to the biological signals and audio signals collected in real time.

[0068] The beneficial effects of the present invention are:

[0069] First, the present invention can monitor the performer's physiological state and micro-movement changes in real time and accurately, extract and analyze physiological signal features through deep learning algorithms, and generate the performer's physiological state vector. This dynamic feedback of physiological state enables the performer to perceive his current physiological state during the performance and make adjustments based on the feedback, thereby achieving the best match between physiology and movement, and effectively improving the performance accuracy and skills.

[0070] Secondly, the present invention combines the performer's micro-movement data with the physiological state, and uses a multimodal self-attention network to perform weighted fusion of various biological signals to further optimize the execution of performance movements. By dynamically adjusting the performer's micro-movements, the system can give precise corrections for each performance movement, helping the performer reduce movement errors and improve the fluency and accuracy of the movement.

[0071] In addition, the present invention uses a simulated annealing algorithm to globally optimize the performer's movements, so that the performance movements gradually converge to the optimal solution. By adjusting performance parameters such as fingering, grip strength, and wrist angle, the performer can achieve more stable and efficient movement control during the performance, reduce unnecessary movement errors, and optimize the performance effect.

[0072] The present invention also dynamically optimizes the sound quality by real-time audio signal acquisition and fast Fourier transform analysis combined with the performer's physiological state vector. The sound quality optimization algorithm can adjust the performance action parameters according to real-time physiological signal feedback, so that the sound quality is consistent with the adjustment of the performance skills, thereby making the performance sound richer and more delicate, meeting the personalized needs of the performer.

[0073] Finally, the present invention uses graph neural networks and Bayesian optimization algorithms to perform collaborative modeling of biological signals and audio signals, establishes an optimization closed loop, and achieves a dynamic balance between the performer's physiological state and sound quality. Through continuous optimization and adjustment, the system can adaptively improve the performer's movement patterns and sound quality characteristics, thereby achieving the best performance effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0075] Figure 1 This is an overall flow chart of a method for optimizing musical instrument playing skills based on biological signal feedback proposed by the present invention;

[0076] Figure 2 Schematic diagram of the graph neural network structure for collaborative modeling of audio signals and biological signals in a method for optimizing musical instrument playing skills based on biological signal feedback proposed in the present invention. DETAILED DESCRIPTION

[0077] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0078] refer to Figure 1 and Figure 2 , a method for optimizing musical instrument playing skills based on biosignal feedback, comprising the following steps:

[0079] S1, collecting multimodal bio-signals of the performer through a sensor array;

[0080] S2, extracting features from the collected multimodal biological signals, identifying physiological features associated with playing skills through a deep convolutional neural network, and generating physiological signal feature vectors;

[0081] S3, based on the generated physiological signal feature vector, using the multimodal self-attention network to perform weighted fusion of various biological signals to generate the performer's physiological state vector;

[0082] S4, using flexible piezoelectric sensors, accelerometers and gyroscopes to collect the player's micro-motion data in real time, monitoring fingertip pressure, wrist rotation angle and finger sliding on the keys;

[0083] S5, based on the collected micro-motion data, establish a motion error model through a support vector machine, analyze the performer's motion error, and calculate the error correction amount;

[0084] S6, optimizing the player's micro-movements using a simulated annealing algorithm according to the error correction amount, and dynamically adjusting the performance action parameters through global optimization, wherein the performance action parameters include fingering, grip strength, and wrist angle;

[0085] S7, through the adjusted performance action parameters, real-time collection of performance audio, and analysis of audio signals using fast Fourier transform, and dynamic optimization of sound quality in combination with the performer's physiological state vector;

[0086] S8. Use graph neural networks and Bayesian optimization algorithms to collaboratively model biological signals and audio signals, establish an optimization closed loop, and adjust the performer's movement pattern and sound quality in real time.

[0087] In this embodiment, the multimodal bio-signals specifically include electromyography, electroencephalography, heart rate and galvanic skin response, which are used to reflect muscle fatigue, neural response, emotional fluctuations and concentration during the performance.

[0088] In this implementation, S3 specifically includes:

[0089] S31, receiving the generated physiological signal feature vector and converting it into a unified format, wherein the physiological signal feature vector includes multimodal biological signal data from the performer, wherein each signal type represents a different physiological feature of the performer during the performance;

[0090] S32. Use the multimodal self-attention network to perform multi-angle fusion calculations on different signal modalities, perform linear transformation on the physiological signal feature vector, obtain the query vector, key vector and value vector, and introduce a multi-head attention mechanism to calculate different weighted values ​​of multiple signal modalities:

[0091]

[0092] Among them, α i,h represents the hth head attention weight of the i-th biological signal, exp represents the exponential function, N represents the total category of biological signals, and d h represents the dimension of the h-th attention head, Q i represents the query vector of the i-th biological signal, K i represents the key vector of the i-th biological signal, K j represents the key vector of the j-th biological signal;

[0093] S33, using the calculated attention weight α i,h The characteristic vectors of various physiological signals are weighted to generate weighted signal characteristic vectors, and a position encoding mechanism is introduced to process the positional relationship between different signals:

[0094]

[0095] Among them, V fused represents the weighted physiological signal feature vector, H represents the number of attention heads, and X i represents the feature vector of the i-th biological signal, P i Represents the positional encoding of the i-th type of biological signal;

[0096] S34, by weighting the physiological signal feature vector V fusedPerform nonlinear transformation and apply deep neural network to perform high-order feature mapping on the signal:

[0097] V transformed =σ(W f ·V fused +b f );

[0098] Among them, V transformed represents the transformed physiological signal feature vector, σ represents the activation function, W f represents the weight matrix, b f represents the bias term;

[0099] S35, the transformed physiological signal feature vector V transformed Combined with the performer's current performance action parameter vector, the association between physiological signals and action parameters is strengthened through a joint embedding model:

[0100] V final =V transformed ⊕A;

[0101] Among them, ⊕ represents the vector concatenation operation, V final represents the final fused physiological state vector, and A represents the performer's current performance action parameter vector;

[0102] S36, through the last layer of the fully connected network, the final fused physiological state vector V final Further optimization and adjustment are performed to generate the performer's physiological state vector.

[0103] In this implementation manner, S5 specifically includes:

[0104] S51, extract features from the collected micro-motion data to obtain the feature vector F of each action features =[f1,f2,…,f m ], where each feature f i Represents real-time data of the performer’s micro-movements during performance;

[0105] S52. Use feature selection methods to select a feature set with high discrimination from micro-motion data And build a training set in represents the feature vector of the jth sample, M represents the total number of samples, y (j) represents the action error value corresponding to the jth sample, reflecting the difference between the performer's current action and the ideal action;

[0106] S53. Use support vector machine to train the training set T to obtain the optimal classification hyperplane. The optimization goal of support vector machine is to maximize the interval, and the support vector is calculated by Lagrange multiplier method:

[0107]

[0108] Among them, f(F selected ) represents the decision function, δ i represents the Lagrange multiplier, represents the feature vector of the i-th sample, b represents the bias term, y (i) Represents the action error value corresponding to the i-th sample;

[0109] S54, use the trained support vector machine to predict the new micro-motion data, input the newly collected motion features into the support vector machine, and output the current motion error value E i ;

[0110] S55, based on the current action error value E i Constructing the weighted error model E weighted , introduce the weight coefficient ω of each action error i Calculate the weighted error value:

[0111]

[0112] Among them, E weighted represents the weighted error value, represents the variance of the i-th error value, and m represents the total number of error values;

[0113] S56, based on the weighted error value E weighted Calculate the error correction:

[0114] ΔD=K·E weighted +λ·∥E weighted ∥2;

[0115] Where ΔD represents the error correction, K represents the adjustment matrix, λ represents the regularization coefficient, ∥E weighted ∥2 represents the L2 norm of the weighted error.

[0116] In this implementation manner, S6 specifically includes:

[0117] S61, setting initial conditions of the simulated annealing algorithm, wherein the initial conditions include an initial temperature and a temperature drop rate, and setting an initial solution of the performance action parameters to be initial micro-action data of the performer, including fingering, grip strength, and wrist angle;

[0118] S62, generating a new candidate solution according to the error correction amount, wherein the candidate solution is generated by perturbing the current performance action parameters, and the perturbation amplitude is controlled by the current temperature; the perturbation process includes fine-tuning the fingering, grip strength and wrist angle, simulating the optimization action that may occur during the performance of the performer, and the generated candidate solution is represented as a set of new performance action parameters;

[0119] S63, calculating the error between the generated candidate solution and the target action, wherein the error represents the optimization effect of the candidate solution, a smaller error represents that the candidate solution is closer to the target action, and a larger error represents that the candidate solution has a larger room for improvement;

[0120] S64, if the error of the candidate solution is less than the error of the current solution, the candidate solution is accepted as the new current solution; if the error of the candidate solution is greater than the error of the current solution, the candidate solution is accepted according to a certain probability, and the certain probability is affected by the current temperature, and the higher the temperature, the greater the acceptance probability;

[0121] S65. After each round of iteration, the simulated annealing algorithm gradually reduces the temperature according to the set cooling rate. The temperature drop means that the amplitude of the disturbance decreases, and the search process gradually concentrates near the local optimal solution. As the number of iterations increases, the optimization process gradually converges until a solution close to the optimal solution is found.

[0122] S66: When the temperature drops to a preset minimum value or the simulated annealing algorithm iterations reach a maximum number, the simulated annealing algorithm is stopped, and the optimization result is the final performance action parameter.

[0123] In this implementation manner, the S7 specifically includes:

[0124] S71, collecting the performer's audio signal in real time through a microphone array, wherein the duration of the audio signal collection is synchronized with the performance process;

[0125] S72, converting the collected audio signal into a frequency domain signal by fast Fourier transform, obtaining spectrum data in each time period, and extracting amplitude information of the audio signal at different frequencies by fast Fourier transform;

[0126] S73, extracting audio spectrum features of the audio signal based on the spectrum data, the audio spectrum features including spectrum amplitude, fundamental frequency, harmonic components and bandwidth; the spectrum amplitude reflects the intensity change of the audio signal, and the fundamental frequency and harmonic components help to describe the pitch and timbre characteristics of the audio;

[0127] S74, combining the audio spectrum feature with the player's physiological state vector, and performing joint processing through a weighted fusion algorithm, so that each frequency component in the audio signal is associated with the corresponding physiological state;

[0128] S75. Based on the comprehensive audio spectrum characteristics and the performer's physiological state vector, the audio signal is dynamically optimized by a sound quality optimization algorithm, wherein the sound quality optimization algorithm dynamically adjusts the performer's action mode and sound quality characteristics based on the sound quality adjustment parameters fed back in real time:

[0129]

[0130] Among them, Q opt represents the optimized sound quality, p represents the adjusted performance action parameter vector, λ1 and λ2 represent weight coefficients, F(X,p) represents the relationship function between the audio signal spectrum characteristics and the action parameters under the current performance action parameters, T represents the target sound quality characteristics, X represents the spectrum feature vector extracted from the audio signal, G(S,p) represents the relationship function between the physiological state vector and the performance action parameters, S ref represents the ideal physiological state vector, and S represents the performer's physiological state vector.

[0131] In this implementation manner, S8 specifically includes:

[0132] S81. Construct a graph neural network, where the biological signals and audio signals are used as nodes in the graph, and the edges between the nodes represent the relationship between the signals.

[0133] S82, generating a final representation vector of each node by aggregating information of adjacent nodes layer by layer in the graph neural network, wherein the final representation vector includes comprehensive features of the biological signal and the audio signal;

[0134] S83, applying a Bayesian optimization algorithm to optimize the node representation output by the graph neural network, wherein the Bayesian optimization algorithm adjusts the parameters of the graph neural network through Gaussian process regression so that the mapping relationship between the biological signal and the audio signal is optimized;

[0135] S84, dynamically adjusting the performer's action pattern and sound quality parameters under the guidance of the Bayesian optimization algorithm, the Bayesian optimization algorithm derives optimized action parameters by evaluating each possible action pattern;

[0136] S85. Based on the optimization results of the graph neural network and the Bayesian optimization algorithm, the performer's movement pattern and sound quality parameters are adjusted through real-time feedback to form a closed-loop control mechanism; the closed-loop control mechanism dynamically updates the performer's movement pattern according to the biological signals and audio signals collected in real time.

[0137] Embodiment 1:

[0138] In order to verify the feasibility of the present invention in practice, an intermediate piano player was selected as the experimental subject, and the experimental site was set up in a piano practice room equipped with high-precision biosignal monitoring equipment. The player has 3 years of piano playing experience and has a certain foundation in note recognition and playing skills, but still faces the challenge of optimizing sound quality and skills. Specifically, when playing difficult music, the player often encounters problems such as uneven finger force distribution, incoherent fingering, and unstable rhythm, resulting in rough sound quality and difficulty in improving playing speed.

[0139] In order to optimize the performer's playing skills and sound quality, the instrument playing skills optimization method based on biosignal feedback of the present invention was applied to the experiment. First, the performer wore a number of biosignal monitoring devices, including electroencephalogram (EEG) sensors, electromyogram (EMG) sensors, and heart rate sensors, to collect their biosignal data in real time. At the same time, the performer played using a piano equipped with a microphone and an audio capture device, and all audio signals were captured in real time and converted into audio data.

[0140] Subsequently, the audio signal and biosignal data are collaboratively modeled through the graph neural network model of the present invention. The model first extracts features from the audio signal and the biosignal respectively and fuses them to form comprehensive features that are input into the graph neural network. In the graph neural network, each node represents a performance action or biosignal feature, and the node information is aggregated through the network to eventually generate a comprehensive feature vector for optimizing the performer's performance actions and sound quality.

[0141] During the optimization process, based on the player's current bio-signal feedback, the system intelligently adjusts the player's fingering strength, rhythm stability, and continuity between notes, and provides real-time optimization feedback. The player listens to real-time feedback through headphones and adjusts his or her performance strategy. After multiple training sessions, the system can provide accurate personalized optimization solutions based on the player's performance data to help him or her improve his or her performance skills.

[0142] The experiment was conducted for 10 days of training, 2 hours per day. The performance of the performers before and after the training was recorded and analyzed in combination with the changes in biological signals and audio signals.

[0143] Table 1 Performer's biological signal and audio signal optimization training data table

[0144]

[0145] Table 1 above shows the gradual improvement of the various skill parameters of the performers after each training. First, in the first few days of training, the performers' scores for fingering force uniformity, rhythm stability, and note continuity were relatively low, at 45, 50, and 55 points, respectively. This shows that although the performers already had a certain foundation before training, there were still obvious deficiencies in their playing skills, especially in the performance of difficult music, where problems such as uneven fingering and unstable rhythm were more prominent. However, after systematic training, the performers' skills were gradually optimized. After 10 days of training, the scores for fingering force uniformity, rhythm stability, and note continuity were increased to 85, 90, and 95 points, respectively, showing a significant improvement in playing skills. This shows that the real-time application of biosignal feedback effectively helps performers improve the accuracy and fluency of their performances.

[0146] Secondly, the performance improvement in note coherence and rhythm stability shows that bio-signal feedback not only improves the physiological performance control, but also enhances the performance's ability to control the rhythm of the music. During the training process, the score of rhythm stability gradually increased from 50 to 90 points, while the score of note coherence increased from 55 to 95 points, indicating that the performance can better control the connection between notes during the performance, making the overall performance more smooth and coordinated.

[0147] The training effect is also significant in terms of playing speed and sound quality. After 10 days of training, the performer's playing speed increased from the original 2.1 minutes per song to 4.3 minutes per song, an increase of nearly 100%. This change shows that the training method optimized by biosignal feedback not only improves the accuracy of playing skills, but also improves the performer's ability to control speed, helping him to play more complex music. In addition, the sound quality score also increased from 6.2 points to 8.6 points, reflecting that while the performer's technical level has improved, the improvement of sound quality has also been systematically optimized. This improvement is mainly due to the real-time adjustment of the performer's fingering strength and rhythm by biosignal feedback, thereby optimizing the sound effect of the notes.

[0148] Overall, the data in the table reflects that through the refined optimization training method of bio-signal feedback, the performers gradually achieved a comprehensive improvement in their playing skills under the collaborative modeling of audio signals and bio-signals. The sustainability of this optimization effect also shows that this method is not only suitable for beginners, but can also effectively help intermediate performers achieve significant improvements in playing skills in a short period of time.

[0149] Judging from the data results, the effectiveness and repeatability of the experiment have been fully verified, indicating that the biosignal feedback optimization method of the present invention has strong practicality and potential for wide application, and is particularly suitable for the training needs of players of various musical instruments in improving their playing skills and sound quality.

[0150] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for optimizing musical instrument playing skills based on biosignal feedback, characterized in that: The steps include: S1, collecting multimodal bio-signals of the performer through a sensor array; S2, extracting features from the collected multimodal biological signals, identifying physiological features associated with playing skills through a deep convolutional neural network, and generating physiological signal feature vectors; S3, based on the generated physiological signal feature vector, using the multimodal self-attention network to perform weighted fusion of various biological signals to generate the performer's physiological state vector; S4, using flexible piezoelectric sensors, accelerometers and gyroscopes to collect the player's micro-motion data in real time, monitoring fingertip pressure, wrist rotation angle and finger sliding on the keys; S5, based on the collected micro-motion data, establish a motion error model through a support vector machine, analyze the performer's motion error, and calculate the error correction amount; S6, optimizing the player's micro-movements using a simulated annealing algorithm according to the error correction amount, and dynamically adjusting the performance action parameters through global optimization, wherein the performance action parameters include fingering, grip strength, and wrist angle; S7, through the adjusted performance action parameters, real-time collection of performance audio, and analysis of audio signals using fast Fourier transform, and dynamic optimization of sound quality in combination with the performer's physiological state vector; S8. Use graph neural networks and Bayesian optimization algorithms to collaboratively model biological signals and audio signals, establish an optimization closed loop, and adjust the performer's movement pattern and sound quality in real time.

2. The method for optimizing musical instrument playing skills based on biosignal feedback according to claim 1, characterized in that: The multimodal bio-signals specifically include electromyography, electroencephalography, heart rate and galvanic skin response, which are used to reflect muscle fatigue, neural response, emotional fluctuations and concentration during the performance.

3. The method for optimizing musical instrument playing skills based on biosignal feedback according to claim 1, characterized in that: The S3 specifically includes: S31, receiving the generated physiological signal feature vector and converting it into a unified format, wherein the physiological signal feature vector includes multimodal biological signal data from the performer, wherein each signal type represents a different physiological feature of the performer during the performance; S32. Use the multimodal self-attention network to perform multi-angle fusion calculations on different signal modalities, perform linear transformation on the physiological signal feature vector, obtain the query vector, key vector and value vector, and introduce a multi-head attention mechanism to calculate different weighted values ​​of multiple signal modalities: Among them, α i,h represents the hth head attention weight of the i-th biological signal, exp represents the exponential function, N represents the total category of biological signals, and d h represents the dimension of the h-th head’s attention, Q i represents the query vector of the i-th biological signal, K i represents the key vector of the i-th biological signal, K j represents the key vector of the j-th biological signal; S33, using the calculated attention weight α i,h The characteristic vectors of various physiological signals are weighted to generate weighted signal characteristic vectors, and a position encoding mechanism is introduced to process the positional relationship between different signals: Among them, V fused represents the weighted physiological signal feature vector, H represents the number of attention heads, and X i represents the feature vector of the i-th biological signal, P i Represents the positional encoding of the i-th type of biological signal; S34, by weighting the physiological signal feature vector V fused Perform nonlinear transformation and apply deep neural network to perform high-order feature mapping on the signal: V transformed =σ(W f ·V fused +b f ); Among them, V transformed represents the transformed physiological signal feature vector, σ represents the activation function, W f represents the weight matrix, b f represents the bias term; S35, the transformed physiological signal feature vector V transformed Combined with the performer's current performance action parameter vector, the association between physiological signals and action parameters is strengthened through a joint embedding model: in, Represents the vector concatenation operation, V final represents the final fused physiological state vector, and A represents the performer's current performance action parameter vector; S36, through the last layer of the fully connected network, the final fused physiological state vector V final Further optimization and adjustment are performed to generate the performer's physiological state vector.

4. The method for optimizing musical instrument playing skills based on biosignal feedback according to claim 1, characterized in that: The S5 specifically includes: S51, extract features from the collected micro-motion data to obtain the feature vector F of each action features =[f1,f2,…,f m ], where each feature f i Represents real-time data of the performer’s micro-movements during performance; S52. Use feature selection methods to select a feature set with high discrimination from micro-motion data And build a training set in represents the feature vector of the jth sample, M represents the total number of samples, y (j) represents the action error value corresponding to the jth sample, reflecting the difference between the performer's current action and the ideal action; S53. Use support vector machine to train the training set T to obtain the optimal classification hyperplane. The optimization goal of support vector machine is to maximize the interval, and the support vector is calculated by Lagrange multiplier method: Among them, f(F selected ) represents the decision function, δ i represents the Lagrange multiplier, represents the feature vector of the i-th sample, b represents the bias term, y (i) Represents the action error value corresponding to the i-th sample; S54, use the trained support vector machine to predict the new micro-motion data, input the newly collected motion features into the support vector machine, and output the current motion error value E i ; S55, based on the current action error value E i Constructing the weighted error model E weighted , introduce the weight coefficient ω of each action error i Calculate the weighted error value: Among them, E weighted represents the weighted error value, represents the variance of the i-th error value, and m represents the total number of error values; S56, based on the weighted error value E weighted Calculate the error correction: ΔD=K·E weighted +λ·∥E weighted ∥2; Where ΔD represents the error correction, K represents the adjustment matrix, λ represents the regularization coefficient, ∥E weighted ∥2 represents the L2 norm of the weighted error.

5. The method for optimizing musical instrument playing skills based on biosignal feedback according to claim 1, characterized in that: The S6 specifically includes: S61, setting initial conditions of the simulated annealing algorithm, wherein the initial conditions include an initial temperature and a temperature drop rate, and setting an initial solution of the performance action parameters to be initial micro-action data of the performer, including fingering, grip strength, and wrist angle; S62, generating a new candidate solution according to the error correction amount, wherein the candidate solution is generated by perturbing the current performance action parameters, and the perturbation amplitude is controlled by the current temperature; the perturbation process includes fine-tuning the fingering, grip strength and wrist angle, simulating the optimization action that may occur during the performance of the performer, and the generated candidate solution is represented as a set of new performance action parameters; S63, calculating the error between the generated candidate solution and the target action, wherein the error represents the optimization effect of the candidate solution, a smaller error represents that the candidate solution is closer to the target action, and a larger error represents that the candidate solution has a larger room for improvement; S64, if the error of the candidate solution is less than the error of the current solution, the candidate solution is accepted as the new current solution; if the error of the candidate solution is greater than the error of the current solution, the candidate solution is accepted according to a certain probability, and the certain probability is affected by the current temperature, and the higher the temperature, the greater the acceptance probability; S65. After each round of iteration, the simulated annealing algorithm gradually reduces the temperature according to the set cooling rate. The temperature drop means that the amplitude of the disturbance decreases, and the search process gradually concentrates near the local optimal solution. As the number of iterations increases, the optimization process gradually converges until a solution close to the optimal solution is found. S66: When the temperature drops to a preset minimum value or the simulated annealing algorithm iterations reach a maximum number, the simulated annealing algorithm is stopped, and the optimization result is the final performance action parameter.

6. The method for optimizing musical instrument playing skills based on biosignal feedback according to claim 1, characterized in that: The S7 specifically includes: S71, collecting the performer's audio signal in real time through a microphone array, wherein the duration of the audio signal collection is synchronized with the performance process; S72, converting the collected audio signal into a frequency domain signal by fast Fourier transform, obtaining spectrum data in each time period, and extracting amplitude information of the audio signal at different frequencies by fast Fourier transform; S73, extracting audio spectrum features of the audio signal based on the spectrum data, the audio spectrum features including spectrum amplitude, fundamental frequency, harmonic components and bandwidth; the spectrum amplitude reflects the intensity change of the audio signal, and the fundamental frequency and harmonic components help to describe the pitch and timbre characteristics of the audio; S74, combining the audio spectrum feature with the player's physiological state vector, and performing joint processing through a weighted fusion algorithm, so that each frequency component in the audio signal is associated with the corresponding physiological state; S75. Based on the comprehensive audio spectrum characteristics and the performer's physiological state vector, the audio signal is dynamically optimized by a sound quality optimization algorithm, wherein the sound quality optimization algorithm dynamically adjusts the performer's action mode and sound quality characteristics based on the sound quality adjustment parameters fed back in real time: Among them, Q opt represents the optimized sound quality, p represents the adjusted performance action parameter vector, λ1 and λ2 represent weight coefficients, F(X,p) represents the relationship function between the audio signal spectrum characteristics and the action parameters under the current performance action parameters, T represents the target sound quality characteristics, X represents the spectrum feature vector extracted from the audio signal, G(S,p) represents the relationship function between the physiological state vector and the performance action parameters, S ref represents the ideal physiological state vector, and S represents the performer's physiological state vector.

7. The method for optimizing musical instrument playing skills based on biosignal feedback according to claim 1, characterized in that: The S8 specifically includes: S81. Construct a graph neural network, where the biological signals and audio signals are used as nodes in the graph, and the edges between the nodes represent the relationship between the signals. S82, generating a final representation vector of each node by aggregating information of adjacent nodes layer by layer in the graph neural network, wherein the final representation vector includes comprehensive features of the biological signal and the audio signal; S83, applying a Bayesian optimization algorithm to optimize the node representation output by the graph neural network, wherein the Bayesian optimization algorithm adjusts the parameters of the graph neural network through Gaussian process regression so that the mapping relationship between the biological signal and the audio signal is optimized; S84, dynamically adjusting the performer's action pattern and sound quality parameters under the guidance of the Bayesian optimization algorithm, the Bayesian optimization algorithm derives optimized action parameters by evaluating each possible action pattern; S85. Based on the optimization results of the graph neural network and the Bayesian optimization algorithm, the performer's movement pattern and sound quality parameters are adjusted through real-time feedback to form a closed-loop control mechanism; the closed-loop control mechanism dynamically updates the performer's movement pattern according to the biological signals and audio signals collected in real time.

Images