Music playing intelligent evaluation system and method based on multi-mode and man-machine cooperation

Abstract

The invention discloses a music playing intelligent evaluation system and method based on multi-mode and man-machine cooperation, and relates to the technical field of music playing intelligent evaluation, and the method comprises the steps: collecting audio signals, video images and playing motion data in a music playing process, and obtaining a multi-mode information data set; performing feature extraction on the multi-mode information data set, and analyzing the audio signal, the video image and the playing action data by applying a convolutional neural network and a recurrent neural network model to generate a basic feature set; based on the basic feature set, utilizing a deep learning algorithm to identify and analyze emotion content conveyed in the playing process, and outputting an emotion intensity value and an emotion category; analyzing intonation error, rhythm deviation and strength matching degree according to comparison between the standard music score and the playing record, evaluating action normalization of the player in combination with action characteristics extracted from the playing action data, and calculating scores of all dimensions; and adjusting the weight between the objective score and the subjective score according to the application scene.

Description

Technical Field

[0001] The present invention relates to the technical field of music performance intelligent evaluation, and in particular to a music performance intelligent evaluation system and method based on multimodality and human-computer collaboration. Background Art

[0002] Intelligent music performance evaluation is a comprehensive, objective, and automated method for evaluating music performances using modern information technology, particularly artificial intelligence. Therefore, how to utilize advanced technologies to improve the intelligence and security of intelligent music performance evaluation has become a pressing issue.

[0003] In the field of intelligent evaluation of music performance, in traditional music performance evaluation, there are time delays or inconsistencies between different types of performance data, which affects the accuracy and consistency of the overall evaluation. In addition, for technical indicators such as pitch error, rhythm deviation, and force matching, traditional manual measurement methods are inefficient and not accurate enough. At the same time, different application scenarios require different scoring standards and weight configurations, and traditional systems are difficult to adjust flexibly. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides an intelligent music performance evaluation method based on multimodality and human-computer collaboration to solve the problem of time delay or inconsistency between different types of performance data in traditional music performance evaluation, which affects the accuracy and consistency of the overall evaluation. In addition, for technical indicators such as pitch error, rhythm deviation, and force matching, the traditional manual measurement method is inefficient and not accurate enough. At the same time, different application scenarios require different scoring standards and weight configurations, and traditional systems are difficult to adjust flexibly.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a method for intelligently evaluating music performance based on multimodality and human-computer collaboration, which includes:

[0008] Collecting audio signals, video images, and performance movement data during music performance to obtain a multimodal information dataset;

[0009] Extracting features from the multimodal information dataset, parsing the audio signal, video image, and performance action data using a convolutional neural network and a recurrent neural network model to generate a basic feature set;

[0010] Based on the basic feature set, a deep learning algorithm is used to identify and analyze the emotional content conveyed during the performance, and output an emotional intensity value and an emotional category;

[0011] By comparing the standard musical score with the performance recording, we analyze the pitch error, rhythm deviation, and dynamic matching, and combine the movement features extracted from the performance movement data to evaluate the performer's movement standardization and calculate the score of each dimension.

[0012] Adjust the weight between objective and subjective scores based on the application scenario, integrate the scores of the two, and calculate a comprehensive score;

[0013] Generate a visual evaluation report based on the scores of each dimension and the comprehensive score;

[0014] The visual evaluation report includes a comprehensive score, a radar chart of each sub-item score, a list of technical issues, a summary of comments, a heat map of emotional expression, and a recommended practice plan.

[0015] As a preferred embodiment of the multimodal and human-computer collaborative music performance intelligent evaluation method of the present invention, the method of collecting audio signals, video images, and performance action data during the music performance to obtain a multimodal information dataset comprises the following specific steps:

[0016] A high-precision microphone array is used to record live music performances and obtain high-quality audio signals. The microphone array is arranged around the performer to ensure all-round coverage and capture complete audio information.

[0017] A noise reduction algorithm is applied to the original audio data to remove background noise and other interference factors and improve the quality of the audio signal. The expression is as follows:

[0018] Use a high-definition camera to record the performer, focusing on facial expressions and body language to obtain video image data. The camera must be installed in a position that ensures an unobstructed field of view and can capture the performer's full body movements.

[0019] Applying frame synchronization technology to the video image data to ensure that each frame of the image accurately matches the audio signal at the corresponding moment, thereby forming a temporally consistent data stream;

[0020] For piano playing, a pressure sensor is installed under the keyboard to record the pressure changes of each key and obtain the performance action data;

[0021] Applying filtering to the performance action data to remove unnecessary noise and errors;

[0022] The processed audio data, video image data, and performance action data are integrated into a multimodal information dataset S(t). The space-time transformation method is used to synchronize the data. The expression is:

[0023]

[0024] Among them, S(t) is the comprehensive signal strength, A clean (τ) is the function of the audio signal intensity after noise reduction processing over time, V sync (τ) is the function of the brightness of the synchronized video frame over time, P filtered (τ) is the function of the performance action data after filtering and changes with time, e -j2πf(t-τ) is the complex exponential function used for Fourier transform, f is the frequency variable, and j is the imaginary unit.

[0025] As a preferred embodiment of the multimodal and human-computer collaborative music performance intelligent evaluation method of the present invention, the following steps are performed: extracting features from the multimodal information dataset, parsing the audio signal, video image, and performance action data by applying a convolutional neural network and a recurrent neural network model, and generating a basic feature set.

[0026] The convolutional neural network CNN is used to synchronize the audio signal A clean (T) Perform spectrum analysis to extract acoustic features of pitch, rhythm, and dynamics. The convolutional neural network structure contains multiple convolutional layers and pooling layers to capture local features in the audio signal. The expression is:

[0027] F a (t)=σ(W a *A clean (t)+b a );

[0028] Among them, F a (t) is the extracted audio feature, W a is the convolution kernel weight matrix, * represents the convolution operation, b a is the bias term, σ is the ReLU activation function;

[0029] The synchronized video image data V sync (t) Apply another set of convolutional neural networks to extract visual features of facial expressions and body language. The convolutional neural network structure used is similar to that used for processing audio signals, but is optimized for two-dimensional images. The expression is:

[0030] F v (t)=σ(W v *V sync (t)+b v );

[0031] Among them, F v (t) is the extracted video feature, W v is the convolution kernel weight matrix, * represents the convolution operation, b vis the bias term, σ is the ReLU activation function;

[0032] The recurrent neural network RNN ​​is used to process the filtered performance action data P filtered (t) Perform time series analysis to extract the action features of keystroke speed, force, and duration. The RNN structure used can be one of the LSTM or GRU units to capture time dependencies. The expression is:

[0033] F p (t)=φ(U p ·P filtered (t)+W p ·F p (t-1)+b p );

[0034] Among them, F p (t) is the extracted action feature, U p is the input weight matrix, P filtered (t) is the performance action data after filtering, W p is the hidden layer weight matrix, F p (t-1) is the action feature of the previous moment, b p is the bias term, φ is the Tanh activation function;

[0035] The extracted audio features F a (t), video features F v (t) and action features F p (t) is fused into a comprehensive feature vector F(t), and the feature fusion is performed using the weighted summation method, which is expressed as:

[0036] F(t)=αF a (t)+βF v (t)+γF p (t);

[0037] Among them, F(t) is the comprehensive feature vector, α is the audio feature weight, β is the video feature weight, and γ is the action feature weight.

[0038] As a preferred solution of the multimodal and human-computer collaborative music performance intelligent evaluation method of the present invention, the method uses a deep learning algorithm based on a basic feature set to identify and analyze the emotional content conveyed during the performance and output the emotional intensity value and emotional category. The specific steps are as follows:

[0039] The sentiment classification model is constructed based on the deep neural network of the Transformer architecture, and the comprehensive feature vector F(t) is preliminarily processed to identify the sentiment category. The expression is:

[0040] E c (t)=softmax(W e F(t)+b e );

[0041] Among them, E c (t) is the probability distribution of emotion category, W e is the weight matrix, F(t) is the comprehensive feature vector, b e is the bias term;

[0042] The probability distribution E of the emotion category after the preliminary processing c (t) Apply the attention mechanism to enhance the focus on key features, thereby improving the accuracy of emotion recognition. The attention mechanism is to weight the feature vector by calculating the importance score of each feature. The expression is as follows:

[0043]

[0044] Among them, A(t) is the weighted feature vector, Q is the query vector, K is the key vector, V is the value vector, d k is the dimension of the key vector;

[0045] The feature vector A(t) processed by the attention mechanism is input into the sentiment intensity regression model to predict the sentiment intensity value. The sentiment intensity regression model is a fully connected layer combined with a linear activation function, and the expression is:

[0046] I(t)=W i A(t)+b i ;

[0047] Among them, I(t) is the emotional intensity value, W i is the weight matrix, A(t) is the weighted eigenvector, b i is the bias term;

[0048] Combined with the emotion category probability distribution E c (t) and the emotion intensity value I(t) to generate the emotion recognition result.

[0049] As a preferred embodiment of the multimodal and human-computer collaborative music performance intelligent evaluation method of the present invention, the method includes: analyzing the pitch error, rhythm deviation, and dynamic matching degree based on the comparison between the standard musical score and the performance recording, and evaluating the performer's movement standardization in combination with the movement features extracted from the performance movement data, and calculating the scores of each dimension. The specific steps are as follows:

[0050] The dynamic time warping algorithm DTW is used to align the standard musical score with the performance recording. The dynamic time warping algorithm DTW finds the best matching path by minimizing the cumulative distance.

[0051] Based on the Fast Fourier Transform (FFT), the pitch error analysis is performed on the time-aligned standard musical score and the performance recording to calculate the difference between the actual frequency and the theoretical frequency of each note. The expression is:

[0052] E f (i)=|F th eoretical (i)-F actual (i)|;

[0053] Among them, E f (i) is the pitch error of the i-th note, F th eoretical (i) is the theoretical frequency of the i-th note in the standard musical notation, F actual (i) is the measured frequency of the i-th note in the actual performance;

[0054] A rhythm deviation analysis is performed on the time-aligned standard musical score and the performance recording to calculate the difference between the actual arrival time and the theoretical arrival time of each beat point. The expression is:

[0055] E r (i)=|T th eoretical (i)-T actual (i)|;

[0056] Among them, E r (i) is the rhythm deviation of the i-th beat point, T th eoretical (i) is the theoretical arrival time of the i-th beat point in the standard musical score, T actual (i) is the arrival time of the i-th beat point in the actual performance;

[0057] A velocity matching analysis is performed on the time-aligned standard musical score and the performance recording to calculate the difference between the actual velocity and the theoretical velocity of each note. The expression is:

[0058] E l (i)=|L th eoretical (i)-L actual (i)|;

[0059] Among them, E l (i) is the intensity matching error of the i-th note, L th eoretical (i) is the theoretical strength of the i-th note in the standard musical notation, L actual (i) is the measured velocity of the i-th note in the actual performance;

[0060] Combined with the movement features extracted from the performance movement data, the performer's movement standardization is evaluated;

[0061] The pitch error E is calculated by weighted summation. f (i) Rhythm deviation Er (i) Force matching error E l (i) and the movement standardization score are integrated into the comprehensive technical score.

[0062] As a preferred solution of the multimodal and human-computer collaborative music performance intelligent evaluation method of the present invention, wherein: the weight between the objective score and the subjective score is adjusted according to the application scenario, the scores of the two are integrated, and a comprehensive score is calculated. The specific steps are:

[0063] Using a predefined application scenario classification model to identify the application scenario of the current performance;

[0064] According to the identified application scenario category CC, the corresponding objective scoring weight and subjective scoring weight are obtained from the preset weight configuration table;

[0065] Standardizing the objective and subjective scores;

[0066] The standardized objective and subjective scores are weighted and summed according to the weights determined by the application scenario to calculate the comprehensive score, which is expressed as:

[0067] S f (t) = w o (C)·S t,norm (t)+w s (C)·S e,norm (t);

[0068] Among them, S f (t) is the comprehensive score, w o (C) is the objective scoring weight, S t,norm (t) is the standardized objective score, w s (C) is the subjective scoring weight, S e,norm (t) is the subjective score after standardization;

[0069] The comprehensive score S f (t) is converted to percentage form, the expression is:

[0070]

[0071] Among them, S r (t) is the final percentage comprehensive score, min(S f ) is the minimum value of the comprehensive score, max(S f ) is the maximum value of the comprehensive score.

[0072] As a preferred solution of the multimodal and human-computer collaborative music performance intelligent evaluation method of the present invention, wherein: the visual evaluation report is generated based on the scores of each dimension and the comprehensive score, and the specific steps are:

[0073] The standardized objective scores, subjective scores, and comprehensive scores are integrated to extract key indicators for visualization. The aggregation function in the extraction process uses a combination of weighted average and maximum value selection, and the expression is:

[0074]

[0075] Among them, K is the key indicator set, max(S t,norm ) is the maximum value of the objective score after standardization, max(S e,norm ) is the maximum value of the subjective score after standardization, ∑w i S i is the weighted score sum, ∑w i is the sum of weights;

[0076] A radar chart mapping function is applied to the key indicator set K to convert it into graphical data that can be intuitively displayed on a two-dimensional plane. The mapping function is used to map each dimension score to the corresponding angular position in the polar coordinate system. The expression is:

[0077] R(θ j )=r j ·(cos θ j ,sin θ j );

[0078] Among them, R(θ j ) is the coordinate point of the jth dimension on the radar chart, r j is the normalized score of the jth dimension, θ j is the angle value corresponding to the dimension in the polar coordinate system;

[0079] Combine emotion recognition results with technical scores to generate an emotion expression heat map;

[0080] A natural language generation model (NLG) is built based on the Transformer structure, and a list of technical questions, a summary of expert comments, and recommended practice plans are automatically generated. The expression is:

[0081] T=decode(W t ·[S t,norm ,E f ]);

[0082] Among them, T is the generated text content, decode(.) is the corresponding character) is the decoding function, W t Generate weight matrix for text, [S t,norm ,E f ] is the corresponding character) is the concatenated score and sentiment feature vector;

[0083] Finally, the radar chart, heat map, text feedback and percentage-based comprehensive scores are integrated into a structured visual evaluation report.

[0084] In a second aspect, the present invention provides an intelligent music performance evaluation system based on multimodality and human-computer collaboration, comprising:

[0085] Data acquisition module, feature extraction module, emotion recognition module, technical evaluation module, comprehensive scoring module and visualization output module;

[0086] The data acquisition module is used to collect audio signals, video images and performance movement data during the music performance, and synchronize multimodal data through space-time transformation to integrate them into a unified time-aligned multimodal information data set;

[0087] The feature extraction module is used to apply convolutional neural networks (CNN) and recurrent neural networks (RNN) to process audio, video, and motion data, extract key features of each modality, and generate a unified basic feature vector through weighted fusion.

[0088] The emotion recognition module is used to build a classification model based on the Transformer architecture, combine the attention mechanism to enhance the weight of key features, identify the emotion category conveyed in the performance, and predict the emotion intensity value through the regression model to form the emotion recognition result;

[0089] The technical evaluation module is used to time-align the performance recording with the standard musical score based on the Dynamic Time Warping algorithm (DTW), analyze the pitch error, rhythm deviation, and dynamic matching, and evaluate the standardization of the movements based on the movement characteristics to generate a multi-dimensional technical score.

[0090] The comprehensive scoring module is used to adaptively adjust the fusion weight between the objective score and the subjective score according to the application scenario classification results, standardize the scores of the two, add them up, and convert them into a percentage system to obtain the final comprehensive scoring result;

[0091] The visualization output module is used to integrate the scores of each dimension and the comprehensive score into a structured visualization report.

[0092] The beneficial effects of the present invention are as follows: by arranging a high-precision microphone array to collect omnidirectional audio signals, the complete capture of acoustic details such as performance timbre and dynamic range is achieved, the comprehensiveness and accuracy of audio collection is improved, thereby improving the reliability of subsequent pitch and rhythm analysis; by using a convolutional neural network to perform spectral analysis on the audio signal, the automatic extraction of key acoustic features such as pitch, rhythm, and dynamics is achieved, the subjective bias brought by manually designed features is avoided, and the generalization ability of the model is improved; by combining a fully connected layer with a linear activation function to construct an emotion intensity regression model, the quantitative output of the degree of emotional expression is achieved, so that emotion recognition not only stays at the category level, but also has an interpretable numerical measurement; by using a dynamic time warping algorithm to time-align the standard musical score and the performance recording, the optimal match between the two on the time axis is achieved, the error and misjudgment problem caused by inconsistent performance speed is overcome, and the scientific nature of the comparative analysis is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0094] Figure 1 A flowchart of an intelligent music performance evaluation method based on multimodality and human-computer collaboration;

[0095] Figure 2 Schematic diagram of an intelligent music performance evaluation system based on multimodality and human-computer collaboration. DETAILED DESCRIPTION

[0096] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0097] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0098] Reference Figure 1 and Figure 2 , provides an intelligent evaluation method for music performance based on multimodality and human-computer collaboration, including the following steps:

[0099] S1, collecting audio signals, video images and performance movement data during the music performance to obtain a multimodal information data set;

[0100] Furthermore, a high-precision microphone array is used to record live music performances to obtain high-quality audio signals. The microphone array is arranged around the performer to ensure all-round coverage to capture complete audio information.

[0101] Apply a noise reduction algorithm to the original audio data to remove background noise and other interference factors and improve the quality of the audio signal. The expression is as follows:

[0102] Use a high-definition camera to record the performer, focusing on facial expressions and body language to obtain video image data. The camera must be installed in a position that ensures an unobstructed field of view and can capture the performer's full-body movements.

[0103] Apply frame synchronization technology to video image data to ensure that each frame of image accurately matches the audio signal at the corresponding moment, forming a temporally consistent data stream;

[0104] For piano playing, a pressure sensor is installed under the keyboard to record the pressure changes of each key and obtain the performance action data;

[0105] Apply filtering to the performance action data to remove unnecessary noise and errors;

[0106] The processed audio data, video image data, and performance action data are integrated into a multimodal information dataset S(t). The space-time transformation method is used to synchronize the data. The expression is:

[0107]

[0108] Among them, S(t) is the comprehensive signal strength, A clean (τ) is the function of the audio signal intensity after noise reduction processing over time, V sync (τ) is the function of the brightness of the synchronized video frame over time, P filtered (τ) is the function of the performance action data after filtering and changes with time, e -j2πf(t-τ) is a complex exponential function used for Fourier transform, f is the frequency variable, and j is the imaginary unit;

[0109] It should be noted that the above-mentioned space-time transformation method is constructed based on Fourier transform, which can effectively realize the fusion expression of multimodal data on a unified time axis. The method not only takes into account the temporal consistency of each modal signal, but also enhances the structural correspondence between different modes through frequency domain analysis. It is suitable for the synchronous processing of multi-source information in complex performance scenarios. In addition, the formula is flexible in form and parameters can be adjusted according to the specific instrument type or the characteristics of the acquisition equipment. It has good versatility and scalability.

[0110] S2. Extract features from the multimodal information dataset by applying convolutional neural network and recurrent neural network models to analyze audio signals, video images, and performance movement data to generate a basic feature set;

[0111] Furthermore, a convolutional neural network (CNN) is used to train the synchronized audio signal A clean (t) Perform spectrum analysis to extract acoustic features of pitch, rhythm, and dynamics. The convolutional neural network structure contains multiple convolutional layers and pooling layers to capture local features in the audio signal. The expression is:

[0112] F a (t)=σ(W a *A clean (t)+b a );

[0113] Among them, F a (t) is the extracted audio feature, W a is the convolution kernel weight matrix, * represents the convolution operation, b a is the bias term, σ is the ReLU activation function;

[0114] The synchronized video image data V sync (t) Apply another set of convolutional neural networks to extract visual features of facial expressions and body language. The convolutional neural network structure used is similar to that used for processing audio signals, but is optimized for two-dimensional images. The expression is:

[0115] F v (t)=σ(W v *V sync (t)+b v );

[0116] Among them, F v (t) is the extracted video feature, W v is the convolution kernel weight matrix, * represents the convolution operation, b v is the bias term, σ is the ReLU activation function;

[0117] The recurrent neural network RNN ​​is used to process the filtered performance action data P filtered (t) Perform time series analysis to extract the action features of keystroke speed, force, and duration. The RNN structure used can be one of the LSTM or GRU units to capture time dependencies. The expression is:

[0118] F p (t)=φ(U p ·P filtered (t)+W p ·F p(t-1)+b p );

[0119] Among them, F p (t) is the extracted action feature, U p is the input weight matrix, P filtered (t) is the performance action data after filtering, W p is the hidden layer weight matrix, F p (t-1) is the action feature of the previous moment, b p is the bias term, φ is the Tanh activation function;

[0120] The extracted audio features F a (t), video features F v (t) and action features F p (t) is fused into a comprehensive feature vector F(t), and the feature fusion is performed using the weighted summation method, which is expressed as:

[0121] F(t)=αF a (t)+βF v (t)+γF p (t);

[0122] Among them, F(t) is the comprehensive feature vector, α is the audio feature weight, β is the video feature weight, and γ is the action feature weight;

[0123] It should be noted that the adoption of a multi-network parallel architecture fully combines the advantages of CNN in local feature extraction and the capabilities of RNN in time series modeling to achieve efficient analysis of audio, video and motion data. The weighted fusion strategy can dynamically adjust the weights of each modality according to task requirements, thereby improving the system's adaptability to different playing styles or instrument types. At the same time, the ReLU and Tanh activation functions used help alleviate the gradient vanishing problem and improve model training efficiency and stability.

[0124] S3. Based on the basic feature set, use deep learning algorithms to identify and analyze the emotional content conveyed during the performance, and output the emotional intensity value and emotional category;

[0125] Furthermore, a sentiment classification model is constructed based on a deep neural network with a Transformer architecture, and the comprehensive feature vector F(t) is preliminarily processed to identify the sentiment category. The expression is:

[0126] E c (t)=softmax(W e F(t)+b e );

[0127] Among them, E c(t) is the probability distribution of emotion category, W e is the weight matrix, F(t) is the comprehensive feature vector, b e is the bias term;

[0128] The probability distribution E of the emotion category after preliminary processing c (t) Apply the attention mechanism to enhance the focus on key features, thereby improving the accuracy of emotion recognition. The attention mechanism is to weight the feature vector by calculating the importance score of each feature. The expression is as follows:

[0129]

[0130] Among them, A(t) is the weighted feature vector, Q is the query vector, K is the key vector, V is the value vector, d k is the dimension of the key vector;

[0131] The feature vector A(t) processed by the attention mechanism is input into the sentiment intensity regression model to predict the sentiment intensity value. The sentiment intensity regression model is a fully connected layer combined with a linear activation function, and the expression is:

[0132] I(t)=W i A(t)+b i ;

[0133] Among them, I(t) is the emotional intensity value, W i is the weight matrix, A(t) is the weighted eigenvector, b i is the bias term;

[0134] Combined with the emotion category probability distribution E c (t) and the emotion intensity value I(t) to generate the emotion recognition result;

[0135] It should be noted that the Transformer architecture combined with the design of the attention mechanism enables the emotion recognition module to have stronger contextual understanding capabilities, which is particularly suitable for capturing the complex patterns of emotion evolution over time during performance. The continuous values ​​output by the emotion intensity regression model can be used as a basis for scoring, helping to generate more personalized emotional feedback. In addition, the emotion recognition results can not only be used to evaluate the performer's expressiveness, but also serve as an important reference for subsequent subjective scoring adjustments, enhancing the artistic perception dimension of the evaluation system.

[0136] S4. Analyze pitch errors, rhythm deviations, and dynamic matching based on a comparison between the standard musical score and the performance recording, and evaluate the performer's movement standardization based on the movement features extracted from the performance movement data, and calculate scores for each dimension;

[0137] Furthermore, the dynamic time warping algorithm DTW is used to align the standard musical score with the performance recording. The dynamic time warping algorithm DTW finds the best matching path by minimizing the cumulative distance.

[0138] Based on the Fast Fourier Transform (FFT), the pitch error analysis of the time-aligned standard musical score and the performance recording is performed to calculate the difference between the actual frequency and the theoretical frequency of each note. The expression is:

[0139] E f (i)=|F th eoretical (i)-F actual (i)|;

[0140] Among them, E f (i) is the pitch error of the i-th note, F th eoretical (i) is the theoretical frequency of the i-th note in the standard musical notation, F actual (i) is the measured frequency of the i-th note in the actual performance;

[0141] Rhythm deviation analysis is performed on the time-aligned standard musical score and performance recording, and the difference between the actual arrival time and the theoretical arrival time of each beat point is calculated. The expression is:

[0142] E r (i)=|T th eoretical (i)-T actual (i)|;

[0143] Among them, E r (i) is the rhythm deviation of the i-th beat point, T th eoretical (i) is the theoretical arrival time of the i-th beat point in the standard musical score, T actual (i) is the arrival time of the i-th beat point in the actual performance;

[0144] The velocity matching analysis of the time-aligned standard musical score and the performance recording is performed to calculate the difference between the actual velocity and the theoretical velocity of each note. The expression is:

[0145] E l (i)=|L th eoretical (i)-L actual (i)|;

[0146] Among them, E l (i) is the intensity matching error of the i-th note, L th eoretical (i) is the theoretical strength of the i-th note in the standard musical notation, L actual (i) is the measured velocity of the i-th note in the actual performance;

[0147] Combined with the movement features extracted from the performance movement data, the performer's movement standardization is evaluated;

[0148] The pitch error E is calculated by weighted summation. f (i) Rhythm deviation E r (i) Force matching error E l (i) and the movement standardization score are integrated into the comprehensive technical score;

[0149] It should be noted that the introduction of the dynamic time warping (DTW) algorithm solves the matching problem caused by inconsistent playing speeds, making error indicators such as pitch and rhythm more comparable. FFT spectrum analysis combined with force energy calculation provides a high-precision basis for technical evaluation. Movement standardization evaluation quantifies playing technique details such as key touch angle and finger coordination through sensor data, providing visual guidance for technical improvement. The comprehensive technical score adopts a weighted summation method, which facilitates the flexible adjustment of the importance of each indicator according to teaching objectives.

[0150] S5. Adjust the weights between the objective and subjective scores based on the application scenario, combine the scores of the two, and calculate a comprehensive score;

[0151] Furthermore, a predefined application scenario classification model is used to identify the application scenario of the current performance;

[0152] According to the identified application scenario category CC, the corresponding objective scoring weight and subjective scoring weight are obtained from the preset weight configuration table;

[0153] Standardize objective and subjective scores;

[0154] The standardized objective and subjective scores are weighted and summed according to the weights determined by the application scenario to calculate the comprehensive score, which is expressed as:

[0155] S f (t) = w o (C)·S t,norm (t)+w s (C)·S e,norm (t);

[0156] Among them, S f (t) is the comprehensive score, w o (C) is the objective scoring weight, S t,norm (t) is the standardized objective score, w s (C) is the subjective scoring weight, S e,norm (t) is the subjective score after standardization;

[0157] The comprehensive score S f(t) is converted to percentage form, the expression is:

[0158]

[0159] Among them, S r (t) is the final percentage comprehensive score, min(S f ) is the minimum value of the comprehensive score, max(S f ) is the maximum value of the comprehensive score;

[0160] It should be noted that the application scenario classification model can automatically adjust the scoring weight configuration according to the actual usage environment to ensure that the evaluation results are more in line with actual needs. The standardization process eliminates the dimensional differences between different scoring dimensions and improves the fairness and consistency of the scoring system. The percentage conversion further enhances the intuitiveness and readability of the scoring results, making it easier for users to quickly understand their own performance level and support horizontal comparison with other performers.

[0161] S6. Generate a visual evaluation report based on the scores of each dimension and the comprehensive score;

[0162] Furthermore, the visual evaluation report includes an overall score, a radar chart of each sub-item score, a list of technical issues, a summary of comments, a heat map of emotional expression, and recommended practice plans;

[0163] The standardized objective scores, subjective scores, and comprehensive scores are integrated to extract key indicators for visualization. The aggregation function in the extraction process uses a combination of weighted average and maximum value selection, and the expression is:

[0164]

[0165] Among them, K is the key indicator set, max(S t,norm ) is the maximum value of the objective score after standardization, max(S e,norm ) is the maximum value of the subjective score after standardization, ∑w i S i is the weighted score sum, ∑w i is the sum of weights;

[0166] Apply the radar chart mapping function to the key indicator set K to convert it into graphical data that can be intuitively displayed on a two-dimensional plane. The mapping function is used to map each dimension score to the corresponding angular position in the polar coordinate system. The expression is:

[0167] R(θ j )=r j ·(cos θ j ,sin θ j );

[0168] Among them, R(θ j ) is the coordinate point of the jth dimension on the radar chart, r j is the normalized score of the jth dimension, θ j is the angle value corresponding to the dimension in the polar coordinate system;

[0169] Combine emotion recognition results with technical scores to generate an emotion expression heat map;

[0170] A natural language generation model (NLG) is built based on the Transformer structure, and a list of technical questions, a summary of expert comments, and recommended practice plans are automatically generated. The expression is:

[0171] T=decode(W t ·[S t,norm ,E f ]);

[0172] Among them, T is the generated text content, decode(.) is the corresponding character) is the decoding function, W t Generate weight matrix for text, [S t,norm ,E f ] is the corresponding character) is the concatenated score and sentiment feature vector;

[0173] Finally, radar charts, heat maps, text feedback, and percentage-based comprehensive scores are integrated into a structured visual evaluation report;

[0174] It should be noted that the visual report integrates multiple presentation forms such as radar charts, heat maps, and natural language comments, taking into account the display effects of technical indicators and artistic expression, and meeting the information acquisition preferences of different groups such as teachers, students and parents. The NLG module can automatically generate personalized feedback suggestions based on scores and emotional characteristics, significantly reducing the cost of manual review. The entire report system supports export to PDF or HTML format, which is convenient for archiving, sharing and continuous tracking of learning progress.

[0175] This embodiment also provides a music performance intelligent evaluation system based on multimodality and human-computer collaboration, including:

[0176] Data acquisition module, feature extraction module, emotion recognition module, technical evaluation module, comprehensive scoring module and visualization output module;

[0177] The data acquisition module is used to collect audio signals, video images, and performance movement data during the music performance, and synchronize multimodal data through space-time transformation to integrate them into a unified time-aligned multimodal information dataset;

[0178] The feature extraction module is used to apply convolutional neural networks (CNN) and recurrent neural networks (RNN) to process audio, video, and motion data, extract key features of each modality, and generate a unified basic feature vector through weighted fusion.

[0179] The emotion recognition module is used to build a classification model based on the Transformer architecture, combine the attention mechanism to enhance the weight of key features, identify the emotion categories conveyed in the performance, and predict the emotion intensity value through the regression model to form the emotion recognition result;

[0180] The technical evaluation module is used to time-align the performance recording with the standard musical score using the Dynamic Time Warping (DTW) algorithm, analyze pitch errors, rhythm deviations, and dynamic matching, and evaluate the standardization of movements based on movement characteristics to generate a multi-dimensional technical score.

[0181] The comprehensive scoring module is used to adaptively adjust the fusion weight between the objective score and the subjective score based on the application scenario classification results, standardize the scores of the two, add them up, and convert them into a percentage system to obtain the final comprehensive score result;

[0182] The visualization output module is used to integrate the scores of each dimension and the comprehensive score into a structured visualization report.

[0183] This embodiment also provides a computer device, which is suitable for the case of an intelligent evaluation method for music performance based on multimodality and human-computer collaboration, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the intelligent evaluation method for music performance based on multimodality and human-computer collaboration proposed in the above embodiment.

[0184] The computer device may be a terminal, comprising a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse.

[0185] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for intelligently evaluating music performance based on multimodality and human-computer collaboration proposed in the above embodiment is implemented. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0186] In summary, the present invention collects omnidirectional audio signals by arranging a high-precision microphone array, thereby achieving complete capture of acoustic details such as performance timbre and dynamic range, improving the comprehensiveness and accuracy of audio collection, and thus improving the reliability of subsequent pitch and rhythm analysis. The audio signal is subjected to spectral analysis by a convolutional neural network, thereby achieving automatic extraction of key acoustic features such as pitch, rhythm, and dynamics, avoiding the subjective bias brought by artificially designed features, and improving the generalization ability of the model. The emotion intensity regression model is constructed by combining a fully connected layer with a linear activation function, thereby achieving quantitative output of the degree of emotional expression, so that emotion recognition not only stays at the category level, but also has an interpretable numerical measurement. The standard music score and the performance recording are time-aligned by a dynamic time warping algorithm, achieving the optimal match between the two on the time axis, overcoming the error and misjudgment problem caused by inconsistent performance speed, and improving the scientific nature of comparative analysis.

[0187] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An intelligent music performance evaluation method based on multimodality and human-computer collaboration, characterized by: include: Collecting audio signals, video images, and performance movement data during music performance to obtain a multimodal information dataset; Extracting features from the multimodal information dataset, parsing the audio signal, video image, and performance action data using a convolutional neural network and a recurrent neural network model to generate a basic feature set; Based on the basic feature set, a deep learning algorithm is used to identify and analyze the emotional content conveyed during the performance, and output an emotional intensity value and an emotional category; By comparing the standard musical score with the performance recording, we analyze the pitch error, rhythm deviation, and dynamic matching, and combine the movement features extracted from the performance movement data to evaluate the performer's movement standardization and calculate the score of each dimension. Adjust the weight between objective and subjective scores based on the application scenario, integrate the scores of the two, and calculate a comprehensive score; Generate a visual evaluation report based on the scores of each dimension and the comprehensive score; The visual evaluation report includes a comprehensive score, a radar chart of each sub-item score, a list of technical issues, a summary of comments, a heat map of emotional expression, and a recommended practice plan.

2. The method for intelligently evaluating music performance based on multimodality and human-computer collaboration as claimed in claim 1, wherein: The method of collecting audio signals, video images and performance action data during the music performance to obtain a multi-modal information data set comprises the following specific steps: A high-precision microphone array is used to record live music performances and obtain high-quality audio signals. The microphone array is arranged around the performer to ensure all-round coverage and capture complete audio information. A noise reduction algorithm is applied to the original audio data to remove background noise and other interference factors and improve the quality of the audio signal. The expression is as follows: Use a high-definition camera to record the performer, focusing on facial expressions and body language to obtain video image data. The camera must be installed in a position that ensures an unobstructed field of view and can capture the performer's full body movements. Applying frame synchronization technology to the video image data to ensure that each frame of the image accurately matches the audio signal at the corresponding moment, thereby forming a temporally consistent data stream; For piano playing, a pressure sensor is installed under the keyboard to record the pressure changes of each key and obtain the performance action data; Applying filtering to the performance action data to remove unnecessary noise and errors; The processed audio data, video image data, and performance action data are integrated into a multimodal information dataset S(t). The space-time transformation method is used to synchronize the data. The expression is: Among them, S(t) is the comprehensive signal strength, A clean (τ) is the function of the audio signal intensity after noise reduction processing over time, V sync (τ) is the function of the brightness of the synchronized video frame over time, P filtered (τ) is the function of the performance action data after filtering and changes with time, e -j2πf(t-τ) is the complex exponential function used for Fourier transform, f is the frequency variable, and j is the imaginary unit.

3. The method for intelligently evaluating music performance based on multimodality and human-computer collaboration as claimed in claim 2, wherein: The feature extraction of the multimodal information dataset is performed by applying convolutional neural network and recurrent neural network models to analyze the audio signal, video image and performance action data to generate a basic feature set. The specific steps are as follows: The convolutional neural network CNN is used to synchronize the audio signal A clean (t) Perform spectrum analysis to extract acoustic features of pitch, rhythm, and dynamics. The convolutional neural network structure contains multiple convolutional layers and pooling layers to capture local features in the audio signal. The expression is: F a (t)=σ(W a *A clean (t)+b a ); Among them, F a (t) is the extracted audio feature, W a is the convolution kernel weight matrix, * represents the convolution operation, b a is the bias term, σ is the ReLU activation function; The synchronized video image data V sync (t) Apply another set of convolutional neural networks to extract visual features of facial expressions and body language. The convolutional neural network structure used is similar to that used for processing audio signals, but is optimized for two-dimensional images. The expression is: F v (t)=σ(W v *V sync (t)+b v ); Among them, F v (t) is the extracted video feature, W v is the convolution kernel weight matrix, * represents the convolution operation, b v is the bias term, σ is the ReLU activation function; The recurrent neural network RNN ​​is used to process the filtered performance action data P filtered (t) Perform time series analysis to extract the action features of keystroke speed, force, and duration. The RNN structure used can be one of the LSTM or GRU units to capture time dependencies. The expression is: F p (t)=φ(U p ·P filtered (t)+W p ·F p (t-1)+b p ); Among them, F p (t) is the extracted action feature, U p is the input weight matrix, P filtered (t) is the performance action data after filtering, W p is the hidden layer weight matrix, F p (t-1) is the action feature of the previous moment, b p is the bias term, φ is the Tanh activation function; The extracted audio features F a (t), video features F v (t) and action features F p (t) is fused into a comprehensive feature vector F(T), and the feature fusion is performed using the weighted summation method, which is expressed as: F(t)=αF a (t)+βF v (t)+γF p (t); Among them, F(t) is the comprehensive feature vector, α is the audio feature weight, β is the video feature weight, and γ is the action feature weight.

4. The method for intelligently evaluating music performance based on multimodality and human-computer collaboration as claimed in claim 3, wherein: Based on the basic feature set, the deep learning algorithm is used to identify and analyze the emotional content conveyed during the performance, and the emotional intensity value and emotional category are output. The specific steps are as follows: The sentiment classification model is constructed based on the deep neural network of the Transformer architecture, and the comprehensive feature vector F(t) is preliminarily processed to identify the sentiment category. The expression is: E c (t)=softmax(W e F(t)+b e ); Among them, E c (t) is the probability distribution of emotion category, W e is the weight matrix, F(t) is the comprehensive feature vector, b e is the bias term; The probability distribution E of the emotion category after the preliminary processing c (t) Apply the attention mechanism to enhance the focus on key features, thereby improving the accuracy of emotion recognition. The attention mechanism is to weight the feature vector by calculating the importance score of each feature. The expression is as follows: Among them, A(t) is the weighted feature vector, Q is the query vector, K is the key vector, V is the value vector, d k is the dimension of the key vector; The feature vector A(t) processed by the attention mechanism is input into the sentiment intensity regression model to predict the sentiment intensity value. The sentiment intensity regression model is a fully connected layer combined with a linear activation function, and the expression is: I(t)=W i A(t)+b i ; Among them, I(t) is the emotional intensity value, W i is the weight matrix, A(T) is the weighted eigenvector, b i is the bias term; Combined with the emotion category probability distribution E c (T) and the emotion intensity value I(t) to generate the emotion recognition result.

5. The method for intelligently evaluating music performance based on multimodality and human-computer collaboration according to claim 4, wherein: The method analyzes pitch error, rhythm deviation, and dynamic matching based on the comparison between the standard musical score and the performance recording, and evaluates the performer's movement standardization in combination with the movement features extracted from the performance movement data, and calculates the scores of each dimension. The specific steps are as follows: The dynamic time warping algorithm DTW is used to align the standard musical score with the performance recording. The dynamic time warping algorithm DTW finds the best matching path by minimizing the cumulative distance. Based on the Fast Fourier Transform (FFT), the pitch error analysis is performed on the time-aligned standard musical score and the performance recording to calculate the difference between the actual frequency and the theoretical frequency of each note. The expression is: E f (i)=|F th eoretical (i)-F actual (i)|; Among them, E f (i) is the pitch error of the i-th note, F th eoretical (i) is the theoretical frequency of the i-th note in the standard musical notation, F actual (i) is the measured frequency of the i-th note in the actual performance; A rhythm deviation analysis is performed on the time-aligned standard musical score and the performance recording to calculate the difference between the actual arrival time and the theoretical arrival time of each beat point. The expression is: E r (i)=|T th eoretical (i)-T actual (i)|; Among them, E r (i) is the rhythm deviation of the i-th beat point, T th eoretical (i) is the theoretical arrival time of the i-th beat point in the standard musical score, T actual (i) is the arrival time of the i-th beat point in the actual performance; A velocity matching analysis is performed on the time-aligned standard musical score and the performance recording to calculate the difference between the actual velocity and the theoretical velocity of each note. The expression is: E l (i)=|L th eoretical (i)-L actual (i)|; Among them, E l (i) is the intensity matching error of the i-th note, L th eoretical (i) is the theoretical strength of the i-th note in the standard musical notation, L actual (i) is the measured velocity of the i-th note in the actual performance; Combined with the movement features extracted from the performance movement data, the performer's movement standardization is evaluated; The pitch error E is calculated by weighted summation. f (i) Rhythm deviation E r (i) Force matching error E l (i) and the movement standardization score are integrated into the comprehensive technical score.

6. The method for intelligently evaluating music performance based on multimodality and human-computer collaboration according to claim 5, wherein: The weights between the objective and subjective scores are adjusted according to the application scenario, the scores of the two are integrated, and a comprehensive score is calculated. The specific steps are as follows: Using a predefined application scenario classification model to identify the application scenario of the current performance; According to the identified application scenario category CC, the corresponding objective scoring weight and subjective scoring weight are obtained from the preset weight configuration table; Standardizing the objective and subjective scores; The standardized objective and subjective scores are weighted and summed according to the weights determined by the application scenario to calculate the comprehensive score, which is expressed as: S f (t)=w o (C)·S t,norm (t)+w s (C)·S e,norm (t); Among them, S f (t) is the comprehensive score, w o (C) is the objective scoring weight, S t,norm (t) is the standardized objective score, w s (C) is the subjective scoring weight, S e,norm (t) is the subjective score after standardization; The comprehensive score S f (t) is converted to percentage form, the expression is: Among them, S r (t) is the final percentage comprehensive score, min(S f ) is the minimum value of the comprehensive score, max(S f ) is the maximum value of the comprehensive score.

7. The method for intelligently evaluating music performance based on multimodality and human-computer collaboration according to claim 6, wherein: The visual evaluation report is generated based on the scores of each dimension and the comprehensive score. The specific steps are as follows: The standardized objective scores, subjective scores, and comprehensive scores are integrated to extract key indicators for visualization. The aggregation function in the extraction process uses a combination of weighted average and maximum value selection, and the expression is: Among them, K is the key indicator set, max(S t,norm ) is the maximum value of the objective score after standardization, max(S e,norm ) is the maximum value of the subjective score after standardization, ∑w i S i is the weighted score sum, ∑w i is the sum of weights; A radar chart mapping function is applied to the key indicator set K to convert it into graphical data that can be intuitively displayed on a two-dimensional plane. The mapping function is used to map each dimension score to the corresponding angular position in the polar coordinate system. The expression is: R(θ j )=r j ·(cos θ j ,sin θ j ); Among them, R(θ j ) is the coordinate point of the jth dimension on the radar chart, r j is the normalized score of the jth dimension, θ j is the angle value corresponding to the dimension in the polar coordinate system; Combine emotion recognition results with technical scores to generate an emotion expression heat map; A natural language generation model (NLG) is built based on the Transformer structure, and a list of technical questions, a summary of expert comments, and recommended practice plans are automatically generated. The expression is: T=decode(W t ·[S t,norm ,E f ]); Among them, T is the generated text content, decode(.) is the corresponding character) is the decoding function, W t Generate weight matrix for text, [S t,norm ,E f ] is the corresponding character) is the concatenated score and sentiment feature vector; Finally, the radar chart, heat map, text feedback and percentage-based comprehensive scores are integrated into a structured visual evaluation report.

8. A music performance intelligent evaluation system based on multimodal and human-computer collaboration, based on the music performance intelligent evaluation method based on multimodal and human-computer collaboration according to any one of claims 1 to 7, characterized in that: include: Data acquisition module, feature extraction module, emotion recognition module, technical evaluation module, comprehensive scoring module and visualization output module; The data acquisition module is used to collect audio signals, video images and performance movement data during the music performance, and synchronize multimodal data through space-time transformation to integrate them into a unified time-aligned multimodal information data set; The feature extraction module is used to apply convolutional neural networks (CNN) and recurrent neural networks (RNN) to process audio, video, and motion data, extract key features of each modality, and generate a unified basic feature vector through weighted fusion. The emotion recognition module is used to build a classification model based on the Transformer architecture, combine the attention mechanism to enhance the weight of key features, identify the emotion category conveyed in the performance, and predict the emotion intensity value through the regression model to form the emotion recognition result; The technical evaluation module is used to time-align the performance recording with the standard musical score based on the Dynamic Time Warping algorithm (DTW), analyze the pitch error, rhythm deviation, and dynamic matching, and evaluate the standardization of the movements based on the movement characteristics to generate a multi-dimensional technical score. The comprehensive scoring module is used to adaptively adjust the fusion weight between the objective score and the subjective score according to the application scenario classification results, standardize the scores of the two, add them up, and convert them into a percentage system to obtain the final comprehensive scoring result; The visualization output module is used to integrate the scores of each dimension and the comprehensive score into a structured visualization report.

Images