Music melody identification method and device for online education

Abstract

The invention relates to the technical field of education, and particularly discloses a music melody recognition method and device for online education, and the method comprises the steps: S1, obtaining audio data; s2, performing pretreatment; s3, feature extraction; s4, melody matching; s5, melody analysis; s6, data feedback; according to the method, the cosine similarity and the DTW algorithm are combined, the tracks or fragments corresponding to the melody played by the user can be accurately recognized, and high recognition accuracy can be kept even under the condition that time deformation or rhythm change exists.

Description

Technical Field

[0001] The present invention belongs to the field of educational technology, and in particular relates to a music melody recognition method and device for online education. Background Art

[0002] In the field of music education, with the rapid development of online education technology, music melody recognition technology has gradually become an important means to improve teaching effectiveness and students' learning experience.

[0003] Traditional feature extraction methods often focus on a single feature such as pitch or rhythm, ignoring the role of other important features in the melody, resulting in one-sided recognition results and poor recognition effect. Therefore, a music melody recognition method and device for online education are proposed. Summary of the invention

[0004] The purpose of the present invention is to provide a music melody recognition method and device for online education to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A music melody recognition method and device for online education, comprising the following steps:

[0007] S1. Audio data acquisition: collecting audio data of music melody played or hummed by the user;

[0008] S2, preprocessing: denoising, filtering and standardizing the collected audio data;

[0009] S3, feature extraction: extracting key features of music melody from the preprocessed audio data, including but not limited to pitch, rhythm, timbre and intensity;

[0010] S4, melody matching: compare the extracted features with the melody features in the music database, identify the track or segment corresponding to the melody, and consider creating part of the adaptation;

[0011] S5. Melody analysis: Analyze the performance style, emotional expression and creative adaptation of the collected data;

[0012] S6. Data feedback: Based on the recognition results and creativity analysis, provide the recognized music information, performance accuracy score, creativity evaluation and improvement suggestions.

[0013] Preferably, in S4, the specific steps of melody matching are:

[0014] S41, standardizing the extracted key features;

[0015] S42, constructing a music database, and performing feature extraction and standardization processing on each entry;

[0016] S43, set the weight vector W of pitch, rhythm and timbre, the feature vector of the audio to be matched is A = [a1, a2, a3], the feature vector in the music database is B = [b1, b2, b3], calculate the cosine similarity, the specific algorithm is as follows;

[0017]

[0018] Among them, W i is the weight of the i-th feature, A i and B i are the i-th elements of vectors A and B respectively;

[0019] S44, set the feature vector of the audio to be matched to C = [c1, c2, c3, ..., c n ], the feature vector in the music database is D = [d1, d2, d3, ..., d m ], and each feature vector is multidimensional, containing e features. The feature weight vector is G = [g1, g2, g3, ..., g k ], where g k is the weight of the kth feature;

[0020] The element H of the weighted distance matrix H ij represents the weighted distance between C at time point i and D at time point j, and is calculated as follows:

[0021]

[0022] Among them, c ik and d ik They are vector c i and d j The kth element of ;

[0023] Initialize a cumulative distance matrix F, whose size is the same as the weighted distance matrix H, that is, n×m;

[0024] Use the dynamic programming algorithm to find the minimum cumulative distance path from the start point to the end point by filling the cumulative distance matrix H. For each element F in the matrix F ij , and its calculation formula is as follows:

[0025] F ij =H ij +min(F i-1,j-1 ,F i-1,j ,F i,j-1 );

[0026] Among them, Fi-1,j-1 ,F i-1,j ,F i,j-1 are the cumulative distances above, above and to the left of the current point, respectively. The initial condition is F 11 =H 11 ;

[0027] After filling the cumulative distance matrix F, from the lower right element F nm Start backtracking to find the path with the minimum cumulative distance. During the backtracking process, according to the direction of the minimum value in the recursive formula (upward, upper left or left), gradually backtrack to the starting point F 11 ;

[0028] The DTW distance is obtained by summing up all weighted distances along the optimal path. The calculation formula is as follows:

[0029]

[0030] Among them, optima l_path represents the index pair (i, j) of all time points on the optimal path;

[0031] S45. Combine cosine similarity and DTW distance. The specific algorithm is as follows:

[0032] S total =α·tsc+β·DTW;

[0033] Among them, α and β are weight factors for adjusting the contribution of the two similarity metrics, and they satisfy α+β=1.

[0034] Preferably, in S5, the melody analysis specifically includes:

[0035] The Mel-frequency cepstral coefficients are used to analyze the playing style of the audio. The extracted features are compared with the known playing style database through a machine learning model to identify the playing style of the audio.

[0036] Extract features related to emotional expression, use the emotion recognition model to match the extracted emotional features with predefined emotional tags, and identify the emotions expressed in the audio;

[0037] Detect audio for plagiarism or imitation of other works to assess originality.

[0038] Preferably, in S6, the data feedback specifically includes:

[0039] Tell the name of the specific track identified, provide information about the track's composer, and the musical style to which the track belongs;

[0040] Give an overall performance accuracy score to reflect the user's overall level of playing the piece, and score the key details in the performance to indicate the performance in these aspects;

[0041] Evaluate whether the user made creative adaptations during performance;

[0042] Provide specific suggestions for improvement based on technical problems encountered by users during performance.

[0043] A device for realizing online educational music melody recognition, comprising:

[0044] An audio acquisition module, used to receive and acquire audio data of music melody played or hummed by a user;

[0045] The preprocessing module is used to perform denoising, filtering, standardization and time segmentation on the collected audio data;

[0046] A feature extraction module is used to extract key features of music melody from preprocessed audio data;

[0047] A melody matching module is used to compare the extracted features with the melody features in the music database to identify the melody played or hummed by the user;

[0048] The feedback and guidance module is used to provide users with instant feedback and personalized learning guidance based on the recognition results.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] (1) By collecting and analyzing the audio data of users’ performances or humming in real time, this method can provide instant feedback on the users’ performance accuracy and creative performance, thereby enhancing the interaction between users and the music education system and providing customized learning suggestions and repertoire recommendations to meet different learning needs.

[0051] (2) By combining cosine similarity and DTW algorithms, it is possible to accurately identify the song or segment corresponding to the melody played by the user, and maintain a high recognition accuracy even in the presence of time deformation or rhythm changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] Embodiment 1:

[0055] See also Figure 1 As shown, a music melody recognition method for online education includes the following steps:

[0056] S1. Audio data acquisition: collecting audio data of music melody played or hummed by the user;

[0057] S2, preprocessing: denoising, filtering and standardizing the collected audio data;

[0058] Denoising is performed through mean filtering. The specific algorithm is as follows: y[n] = (x[n] + x[n-1] + x[n+1]) / 3. The value of each data point is replaced by the average value of its surrounding data points.

[0059] Calculates the maximum and minimum values ​​of an audio signal.

[0060] Standardization processing, according to the target range, linearly transforms the audio signal so that its amplitude falls within the target range. The audio signal is x[n], the maximum value is x_max, and the minimum value is x_min.

[0061] The normalized signal y[n] can be calculated by the following formula:

[0062]

[0063] S3. Feature extraction: Extract key features of the music melody from the preprocessed audio data, including but not limited to pitch, rhythm, timbre and intensity, as follows:

[0064] The audio signal is divided into multiple short time frames, usually each frame is 20-30 milliseconds long, and there is a certain overlap between frames;

[0065] Apply a window function to each short-time frame to reduce spectrum leakage at the edge of the frame. The window function can make the signals at both ends of the frame transition smoothly to zero, avoiding edge effects in spectrum analysis.

[0066] Perform FFT on each windowed frame to convert the signal from time domain to frequency domain to obtain a spectrum representation;

[0067] According to the FFT results, various audio feature parameters are calculated, such as pitch, rhythm, timbre and intensity. These feature parameters will be used for subsequent music melody recognition and analysis;

[0068] Pitch: Perform autocorrelation calculation on each frame, find the first local maximum in the autocorrelation function, and its corresponding delay is the fundamental frequency period. Convert the fundamental frequency period to pitch (unit: Hz) using the formula f = 1 / T, where f is the pitch and T is the fundamental frequency period.

[0069] Rhythm: Apply beat detection algorithms to identify beat positions in audio signals, analyze the intervals and intensity changes between beats, and extract rhythmic pattern features;

[0070] Timbre: obtained by calculating the spectral centroid, spectral roll-off point, spectral flux, spectral entropy and other parameters of the audio signal; the spectral centroid reflects the energy distribution center of the audio signal spectrum, the spectral roll-off point is the frequency point where the cumulative energy in the spectrum reaches a certain proportion (such as 85%), the spectral flux indicates the degree of change of the spectrum between adjacent frames, and the spectral entropy reflects the complexity or uncertainty of the audio signal spectrum;

[0071] Sound intensity: Calculate the square sum of the signal for each frame, then take the average value to get the short-time energy of the frame, and use the short-time energy to measure the strength of the audio signal;

[0072] S4, melody matching: compare the extracted features with the melody features in the music database, identify the track or segment corresponding to the melody, and consider creating part of the adaptation;

[0073] S5. Melody analysis: Analyze the performance style, emotional expression and creative adaptation of the collected data;

[0074] S6. Data feedback: Based on the recognition results and creativity analysis, provide the recognized music information, performance accuracy score, creativity evaluation and improvement suggestions.

[0075] In one embodiment of the present invention, in S4, the specific steps of melody matching are:

[0076] S41, standardizing the extracted key features;

[0077] S42, constructing a music database, and performing feature extraction and standardization processing on each entry;

[0078] S43, set the weight vector W of pitch, rhythm and timbre, the feature vector of the audio to be matched is A = [a1, a2, a3], the feature vector in the music database is B = [b1, b2, b3], calculate the cosine similarity, the specific algorithm is as follows;

[0079]

[0080] Among them, W i is the weight of the i-th feature, A i and B i are the i-th elements of vectors A and B respectively;

[0081] S44, set the feature vector of the audio to be matched to C = [c1, c2, c3, ..., c n ], the feature vector in the music database is D = [d1, d2, d3, ..., d m ], and each feature vector is multidimensional, containing e features. The feature weight vector is G = [g1, g2, g3, ..., g k ], where g k is the weight of the kth feature;

[0082] The element H of the weighted distance matrix H ij represents the weighted distance between C at time point i and D at time point j, and is calculated as follows:

[0083]

[0084] Among them, c ik and d ik They are vector c i and d j The kth element of ;

[0085] Initialize a cumulative distance matrix F, whose size is the same as the weighted distance matrix H, that is, n×m;

[0086] Use the dynamic programming algorithm to find the minimum cumulative distance path from the start point to the end point by filling the cumulative distance matrix H. For each element F in the matrix F ij , and its calculation formula is as follows:

[0087] F ij =H ij +min(F i-1,j-1 ,F i-1,j ,F i,j-1 );

[0088] Among them, F i-1,j-1 ,F i-1,j ,F i,j-1 are the cumulative distances above, above and to the left of the current point, respectively. The initial condition is F 11 =H 11 ;

[0089] After filling the cumulative distance matrix F, from the lower right element F nmStart backtracking to find the path with the minimum cumulative distance. During the backtracking process, according to the direction of the minimum value in the recursive formula (upward, upper left or left), gradually backtrack to the starting point F 11 ;

[0090] The DTW distance is obtained by summing up all weighted distances along the optimal path. The calculation formula is as follows:

[0091]

[0092] Among them, optima l_path represents the index pair (i, j) of all time points on the optimal path;

[0093] S45. Combine cosine similarity and DTW distance. The specific algorithm is as follows:

[0094] S total =α·tsc+β·DTW;

[0095] Among them, α and β are weight factors for adjusting the contribution of the two similarity metrics, and they satisfy α+β=1.

[0096] In one embodiment of the present invention, in S5, the melody analysis specifically includes:

[0097] The Mel-frequency cepstral coefficients are used to analyze the playing style of the audio. The extracted features are compared with the known playing style database through a machine learning model to identify the playing style of the audio.

[0098] Extract features related to emotional expression, use the emotion recognition model to match the extracted emotional features with predefined emotional tags, and identify the emotions expressed in the audio;

[0099] Detect audio for plagiarism or imitation of other works to assess originality.

[0100] In one embodiment of the present invention, in S6, the data feedback specifically includes:

[0101] Tell the name of the specific track identified, provide information about the track's composer, and the musical style to which the track belongs;

[0102] Give an overall performance accuracy score to reflect the user's overall level of playing the piece, and score the key details in the performance to indicate the performance in these aspects;

[0103] Evaluate whether the user made creative adaptations during performance;

[0104] Provide specific suggestions for improvement based on technical problems encountered by users during performance.

[0105] A device for realizing online education music melody recognition, applicable to the above-mentioned music melody recognition method for online education, comprising:

[0106] An audio acquisition module, used to receive and acquire audio data of music melody played or hummed by a user;

[0107] The preprocessing module is used to perform denoising, filtering, standardization and time segmentation on the collected audio data;

[0108] A feature extraction module is used to extract key features of music melody from preprocessed audio data;

[0109] A melody matching module is used to compare the extracted features with the melody features in the music database to identify the melody played or hummed by the user;

[0110] The feedback and guidance module is used to provide users with instant feedback and personalized learning guidance based on the recognition results.

[0111] As can be seen from the above, the audio data of the music melody played or hummed by the user is received and collected through the audio collection module, which is the starting point of the entire recognition process;

[0112] Perform a series of processing on the collected audio data, including denoising, filtering, standardization and time segmentation, to improve the quality and processability of the audio data;

[0113] Extract key features of music melody from preprocessed audio data, including but not limited to pitch, rhythm, timbre and intensity, which are the basis for subsequent matching and analysis;

[0114] By building a music database, extracting and normalizing features for each entry, and then using cosine similarity and dynamic time warping (DTW) algorithms, the extracted features are compared with the melody features in the music database to identify the track or segment corresponding to the melody played or hummed by the user, while taking into account the creative adaptation content to ensure the accuracy and robustness of the match;

[0115] After identifying the melody, the collected data was further analyzed for performance style, emotional expression, and creative adaptation;

[0116] Provide detailed feedback to users based on recognition results and creativity analysis.

[0117] The standard parts used in the present invention can all be purchased from the market, and the special-shaped parts can be customized according to the description and the drawings. The specific connection methods of each part adopt conventional means such as mature bolts, rivets, welding, etc. in the prior art. The machinery, parts and equipment all adopt conventional models in the prior art, and the circuit connection adopts the conventional connection method in the prior art, which will not be described in detail here. The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.

[0118] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. "Multiple" means two or more, unless otherwise clearly and specifically defined.

[0119] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0120] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0121] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0122] In the drawings of the embodiments disclosed in the present invention, only the structures related to the embodiments disclosed in the present invention are involved, and other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of the present invention can be combined with each other.

[0123] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A music melody recognition method for online education, characterized in that: The following steps are involved: S1. Audio data acquisition: collecting audio data of music melody played or hummed by the user; S2, preprocessing: denoising, filtering and standardizing the collected audio data; S3, feature extraction: extracting key features of music melody from the preprocessed audio data, including but not limited to pitch, rhythm, timbre and intensity; S4, melody matching: compare the extracted features with the melody features in the music database, identify the track or segment corresponding to the melody, and consider creating part of the adaptation; S5. Melody analysis: Analyze the performance style, emotional expression and creative adaptation of the collected data; S6. Data feedback: Based on the recognition results and creativity analysis, provide the recognized music information, performance accuracy score, creativity evaluation and improvement suggestions.

2. A music melody recognition method and device for online education according to claim 1, characterized in that: In S4, the specific steps of melody matching are: S41, standardizing the extracted key features; S42, constructing a music database, and performing feature extraction and standardization processing on each entry; S43, set the weight vector W of pitch, rhythm and timbre, the feature vector of the audio to be matched is A = [a1, a2, a3], the feature vector in the music database is B = [b1, b2, b3], calculate the cosine similarity, the specific algorithm is as follows; Among them, W i is the weight of the i-th feature, A i and B i are the i-th elements of vectors A and B respectively; S44, set the feature vector of the audio to be matched to C = [c1, c2, c3, ..., c n ], the feature vector in the music database is D = [d1, d2, d3, ..., d m ], and each feature vector is multidimensional, containing e features. The feature weight vector is G = [g1, g2, g3, ..., g k ], where g k is the weight of the kth feature; The element H of the weighted distance matrix H ij represents the weighted distance between C at time point i and D at time point j, and is calculated as follows: Among them, c ik and d ik They are vector c i and d j The kth element of ; Initialize a cumulative distance matrix F, whose size is the same as the weighted distance matrix H, that is, n×m; Use the dynamic programming algorithm to find the minimum cumulative distance path from the start point to the end point by filling the cumulative distance matrix H. For each element F in the matrix F ij , and its calculation formula is as follows: F ij =H ij +min(F i-1,j-1 ,F i-1,j ,F i,j-1 ); Among them, F i-1,j-1 ,F i-1,j ,F i,j-1 are the cumulative distances above, above and to the left of the current point, respectively. The initial condition is F 11 =H 11 ; After filling the cumulative distance matrix F, from the lower right element F nm Start backtracking to find the path with the minimum cumulative distance. During the backtracking process, according to the direction of the minimum value in the recursive formula (upward, upper left or left), gradually backtrack to the starting point F 11 ; The DTW distance is obtained by summing up all weighted distances along the optimal path. The calculation formula is as follows: Among them, optimal_path represents the index pairs (i, j) of all time points on the optimal path; S45. Combine cosine similarity and DTW distance. The specific algorithm is as follows: S total =α·tsc+β·DTW; Among them, α and β are weight factors for adjusting the contribution of the two similarity metrics, and they satisfy α+β=1.

3. The method and device for music melody recognition for online education according to claim 1, characterized in that: In S5, the melody analysis specifically includes: The Mel-frequency cepstral coefficients are used to analyze the playing style of the audio. The extracted features are compared with the known playing style database through a machine learning model to identify the playing style of the audio. Extract features related to emotional expression, use the emotion recognition model to match the extracted emotional features with predefined emotional tags, and identify the emotions expressed in the audio; Detect audio for plagiarism or imitation of other works to assess originality.

4. The method and device for music melody recognition for online education according to claim 1, characterized in that: In S6, the data feedback specifically includes: Tell the name of the specific track identified, provide information about the track's composer, and the musical style to which the track belongs; Give an overall performance accuracy score to reflect the user's overall level of playing the piece, and score the key details in the performance to indicate the performance in these aspects; Evaluate whether the user made creative adaptations during performance; Provide specific suggestions for improvement based on technical problems encountered by users during performance.

5. A device for realizing online educational music melody recognition, characterized in that: include: An audio acquisition module, used to receive and acquire audio data of music melody played or hummed by a user; The preprocessing module is used to perform denoising, filtering, standardization and time segmentation on the collected audio data; A feature extraction module is used to extract key features of music melody from preprocessed audio data; A melody matching module is used to compare the extracted features with the melody features in the music database to identify the melody played or hummed by the user; The feedback and guidance module is used to provide users with instant feedback and personalized learning guidance based on the recognition results.

Images