A singing evaluation method based on deep learning

Abstract

The present invention provides a singing evaluation method based on deep learning in the technical field of singing evaluation, comprising the following steps: step S10, obtaining a large amount of singing data, and cleaning each of the singing data; step S20, extracting the audio features of each of the cleaned singing data, and constructing a feature data set; step S30, creating an audio alignment model, and aligning each audio feature in the feature data set based on the audio alignment model; step S40, creating a multidimensional evaluation model based on deep learning, and training the multidimensional evaluation model using the aligned feature data set; step S50, obtaining the audio to be evaluated, inputting the audio to be evaluated into the multidimensional evaluation model, and obtaining and displaying the pitch evaluation value, rhythm evaluation value, and pronunciation evaluation value. The advantages of the present invention are: greatly improving the accuracy and interpretability of singing evaluation.

Description

Technical Field

[0001] The present invention relates to the technical field of singing evaluation, and in particular to a singing evaluation method based on deep learning. Background Art

[0002] As music education gradually receives more attention from the society, the number of people participating in music learning and engaging in music education continues to grow. It is very important to receive timely evaluation and feedback in the initial stage of learning any subject, and music learning is no exception. Music learning is mainly a process of accumulating proficiency. Early recognition of one's own shortcomings can prevent students from developing initial mistakes into deep-rooted and difficult-to-correct bad habits after day-to-day practice. However, in the process of music learning, unless the students grow up in a music family, they cannot always have a teacher by their side, and students cannot receive immediate and professional feedback, which greatly limits the students' learning efficiency.

[0003] Although there are some singing evaluation systems on the market, they are all designed and implemented around entertainment scenarios. The target users in these scenarios do not really care whether the final evaluation results are completely accurate, nor do they care which part is not done well. The evaluation results of singing evaluation often lack accuracy and explainability.

[0004] Therefore, how to provide a singing evaluation method based on deep learning to improve the accuracy and interpretability of singing evaluation has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a singing evaluation method based on deep learning, so as to improve the accuracy and interpretability of singing evaluation.

[0006] The present invention is implemented as follows: a singing evaluation method based on deep learning comprises the following steps:

[0007] Step S10, obtaining a large amount of song data, and cleaning each of the song data;

[0008] Step S20, extracting the cleaned audio features of each of the singing data to construct a feature data set;

[0009] Step S30: creating an audio alignment model, and aligning the audio features in the feature data set based on the audio alignment model;

[0010] Step S40: creating a multi-dimensional evaluation model based on deep learning, and training the multi-dimensional evaluation model using the aligned feature data set;

[0011] Step S50: obtain the audio to be evaluated, input the audio to be evaluated into a multi-dimensional evaluation model, obtain a pitch evaluation value, a rhythm evaluation value, and a pronunciation evaluation value, and display them.

[0012] Furthermore, the step S10 is specifically as follows:

[0013] A large amount of singing data is obtained from different channels, and each of the singing data is sequentially format converted, vocals are separated, blank segments are removed, and noise is reduced.

[0014] Furthermore, the step S20 is specifically as follows:

[0015] The cleaned singing data are preprocessed to obtain corresponding frequency domain signals, and audio features including Mel-frequency cepstral coefficients, chromaticity features, beat graphs, signal-to-noise ratio, harmonic noise ratio, frequency perturbations and resonance peaks are extracted based on the frequency domain signals, and a feature data set is constructed based on the audio features.

[0016] Furthermore, the preprocessing of the cleaned singing data to obtain the corresponding frequency domain signal is specifically:

[0017] The cleaned singing data are pre-emphasized, framed and windowed in turn to obtain corresponding time domain signals, and each time domain signal is fast Fourier transformed to obtain corresponding frequency domain signals, thereby completing the preprocessing of the singing data.

[0018] Furthermore, the Mel-frequency cepstrum coefficients are obtained by performing Mel-frequency filter bank filtering, discrete cosine transform, and dynamic difference parameter extraction on the frequency domain signal.

[0019] Furthermore, the chromaticity feature includes a chromaticity vector and a chromaticity spectrum.

[0020] Furthermore, in step S30, the audio alignment model is created based on Guided Attention.

[0021] Furthermore, in step S40, the multi-dimensional evaluation model includes a pitch evaluation sub-model, a rhythm evaluation sub-model and a pronunciation evaluation sub-model.

[0022] Furthermore, the step S50 is specifically as follows:

[0023] Obtain the audio to be evaluated, clean it, extract audio features and align it in sequence, input it into the pitch evaluation sub-model to obtain a pitch evaluation value, input it into the rhythm evaluation sub-model to obtain a rhythm evaluation value, input it into the pronunciation evaluation sub-model to obtain a pronunciation evaluation value, store and display the pitch evaluation value, rhythm evaluation value and pronunciation evaluation value.

[0024] The advantages of the present invention are:

[0025] The singing data is format converted, voice separated, blank segment removed, and noise reduced in sequence, and then the audio features including Mel-frequency cepstral coefficients, chromaticity features, beat graphs, signal-to-noise ratio, harmonic-to-noise ratio, frequency perturbation, and resonance peaks are extracted to construct a feature data set. The audio features in the feature data set are aligned using the audio alignment model created based on Guided Attention. The created pitch evaluation sub-model, rhythm evaluation sub-model, and pronunciation evaluation sub-model are trained using the aligned feature data set. Finally, the trained pitch evaluation sub-model, rhythm evaluation sub-model, and pronunciation evaluation sub-model are used to perform singing evaluation on the audio to be evaluated, and the corresponding pitch evaluation value, rhythm evaluation value, and pronunciation evaluation value are obtained. Since the multidimensional evaluation model is trained through multidimensional features, and the evaluation values ​​corresponding to each segment of the audio to be evaluated are given based on the three dimensions of pitch, rhythm, and pronunciation, that is, a fine-grained and multi-dimensional evaluation is performed on the audio to be evaluated, the accuracy and interpretability of the singing evaluation are greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0027] Figure 1 It is a flow chart of a singing evaluation method based on deep learning of the present invention. DETAILED DESCRIPTION

[0028] The technical solution in the embodiments of the present application has the following overall idea: a multidimensional evaluation model is trained through the audio features of Mel-frequency cepstral coefficients, chromaticity features, beat diagrams, signal-to-noise ratio, harmonic-to-noise ratio, frequency perturbations, and resonance peaks, and evaluation values ​​corresponding to each audio segment to be evaluated are given based on the three dimensions of pitch, rhythm, and pronunciation, that is, a fine-grained, multi-dimensional evaluation is performed on the audio to be evaluated to improve the accuracy and interpretability of the singing evaluation.

[0029] Please refer to Figure 1 As shown, a preferred embodiment of a singing evaluation method based on deep learning of the present invention comprises the following steps:

[0030] Step S10, obtaining a large amount of song data, and cleaning each of the song data; the song data carries lyrics;

[0031] Step S20, extracting the audio features of each of the cleaned singing data to construct a feature data set; in order to evaluate different singing data, a unified audio feature is required to represent various aspects of the singing data, and different audio features can reflect the singing data from different aspects, and different audio features have different evaluation effects on the singing data, so it is particularly important to extract multi-dimensional and effective audio features;

[0032] Step S30: creating an audio alignment model, and aligning the audio features in the feature data set based on the audio alignment model;

[0033] Step S40: creating a multi-dimensional evaluation model based on deep learning (deep neural network, DNN), and training the multi-dimensional evaluation model using the aligned feature data set; DNN is a computational model composed of a large number of nodes (or neurons) directly interconnected, and the output of the network is different due to different activation functions and weights, which is an approximation of a certain function or an approximate description of a mapping relationship;

[0034] Step S50: obtain the audio to be evaluated, input the audio to be evaluated into a multi-dimensional evaluation model, obtain a pitch evaluation value, a rhythm evaluation value, and a pronunciation evaluation value, and display them.

[0035] The multidimensional evaluation model established through supervised learning can automatically focus on effective feature dimensions according to the labels of the target dimensions, and the multidimensional evaluation model of the present invention performs fragmentary evaluation on singing data. While taking into account multidimensional evaluation, it can also achieve finer-grained evaluation, which can effectively provide feedback to users in educational scenarios.

[0036] The step S10 is specifically as follows:

[0037] A large amount of singing data can be obtained from different channels, and each of the singing data can be format converted, voice separated, blank segment removed, and noise reduced. The singing data can be obtained from music websites, public data sets in the field of music information retrieval, manual collection, and other channels.

[0038] Since the formats of the collected singing data may be varied, and the wav format is a more common, simple and intuitive format, in order to facilitate subsequent unified processing, the format is uniformly converted to wav, and the sampling rate and number of channels of all singing data are unified. Since most of the collected singing data are accompanied by accompaniment or other background noise, in order to make the subsequent model pay more attention to the human voice in the singing data, it is necessary to separate the human voice from the singing data. Since there are blank parts at the beginning, end and middle of ordinary singing data, in order to make the creation of subsequent models more efficient, it is necessary to perform sound time detection and eliminate blank segments. Since the collected singing data may contain noise, obvious noise will have a negative impact on the model, so it is necessary to reduce the noise to a certain threshold.

[0039] The step S20 is specifically as follows:

[0040] The cleaned singing data are preprocessed to obtain the corresponding frequency domain signals, and audio features including Mel frequency cepstrum coefficients, chromaticity features, tempograms, signal-to-noise ratios, harmonic noise ratios, frequency perturbations, and resonance peaks are extracted based on the frequency domain signals, and a feature data set is constructed based on the audio features. The basic method for analyzing music beat intervals is called a tempo graph, which is generally obtained by analyzing a spectrogram. It indicates the size of the music beat interval within a period of time and contains the tempo information that changes over time in the singing audio.

[0041] The preprocessing of the cleaned singing data to obtain the corresponding frequency domain signal is specifically:

[0042] The cleaned singing data are pre-emphasized, framed and windowed in turn to obtain corresponding time domain signals, and each time domain signal is fast Fourier transformed to obtain corresponding frequency domain signals, thereby completing the preprocessing of the singing data.

[0043] The Mel-scale Frequency Cepstral Coefficients (MFCC) are obtained by filtering the frequency domain signal with a Mel filter group, performing discrete cosine transform, and extracting dynamic difference parameters. The Mel-scale frequency cepstral coefficients are cepstral parameters extracted in the Mel-scale frequency domain, taking into account the auditory characteristics of humans, first mapping the linear spectrum to the Mel nonlinear spectrum based on auditory perception, and then converting it to the cepstral. The Mel scale describes the nonlinear characteristics of the human ear frequency.

[0044] The chromaticity features include a chromaticity vector and a chromaticity spectrum. A chromaticity vector is a vector containing 12 elements, which represent the energy of 12 pitches in a time slice (such as 1 frame), that is, the energy accumulation of the same pitch in different octaves; a chromaticity spectrum is a sequence of chromaticity vectors, summarizing the pitch type information in the audio time slice.

[0045] In step S30, the audio alignment model is created based on Guided Attention. Weakly supervised learning with only coarse labels can obtain visual interpretations of deep neural networks by back-propagating gradients, such as attention maps, which can be used as priors for tasks such as object localization and semantic segmentation. In speech scenarios, due to the temporal nature of singing audio, theoretically, the attention map only has valid values ​​within a certain range of the main diagonal. Guided Attention adds this constraint to the Attention mechanism. Once there is a valid value that deviates from the diagonal, a corresponding penalty will be given.

[0046] In step S40, the multi-dimensional evaluation model includes a pitch evaluation sub-model, a rhythm evaluation sub-model and a pronunciation evaluation sub-model.

[0047] From the perspective of pitch, the pitch sung should be consistent with the prescribed pitch, and the evaluation can be calculated by technical observations such as pitch. The traditional method regards singing evaluation as a regression or multi-segment classification task, and uses an end-to-end deep learning model to directly perform supervised learning, but it cannot meet the fine-grained evaluation feedback of primary and secondary education scenarios. The present invention adopts a metric learning model based on fragment input, and models from multiple dimensions based on fragments of singing data. The input of the pitch evaluation submodel includes three channels, namely, the MFCC, chromaticity matrix, and beat map of the reference sample, high-quality sample, and low-quality sample. The high-dimensional features of the input are mapped to a low-dimensional space through a neural network, and through Triplet Loss and back propagation, the pitch evaluation submodel is prompted to divide the high-quality samples into a low-dimensional space with a closer distance, and divide the low-quality samples into a low-dimensional space with a farther distance, and then score each sample based on the distance of the vector in the low-dimensional space.

[0048] From the perspective of rhythm, we should be able to remember the length and combination of notes in the song clearly, express strength and weakness accurately, and express the rhythmic patterns that express the characteristics of the music clearly. This can be achieved through technical observations such as the length, speed, and pauses of the notes. When creating the rhythm evaluation sub-model, in addition to the above-mentioned multiple features, indicators such as the length, speed, and pause degree of each note are also added, and these estimated features are used as important indicators for evaluating the accuracy of the rhythm.

[0049] From the perspective of pronunciation, each word in the lyrics should be clearly sung out when singing, with clear articulation and accurate pronunciation, which can be achieved through technical observations such as lyrics pronunciation, phoneme pronunciation, and quasi-speech recognition. First, the lyrics of the singing data are identified to obtain the probability distribution of the speech sequence, and then the phoneme sequence of the sample to be evaluated is aligned with the standard phoneme sequence based on the audio alignment model, and then a score is given based on the probability distribution of the phoneme to be evaluated.

[0050] The step S50 is specifically as follows:

[0051] Obtain the audio to be evaluated, clean it, extract audio features and align it in sequence, input it into the pitch evaluation sub-model to obtain a pitch evaluation value, input it into the rhythm evaluation sub-model to obtain a rhythm evaluation value, input it into the pronunciation evaluation sub-model to obtain a pronunciation evaluation value, store and display the pitch evaluation value, rhythm evaluation value and pronunciation evaluation value.

[0052] In summary, the advantages of the present invention are:

[0053] The singing data is format converted, voice separated, blank segment removed, and noise reduced in sequence, and then the audio features including Mel-frequency cepstral coefficients, chromaticity features, beat graphs, signal-to-noise ratio, harmonic-to-noise ratio, frequency perturbation, and resonance peaks are extracted to construct a feature data set. The audio features in the feature data set are aligned using the audio alignment model created based on Guided Attention. The created pitch evaluation sub-model, rhythm evaluation sub-model, and pronunciation evaluation sub-model are trained using the aligned feature data set. Finally, the trained pitch evaluation sub-model, rhythm evaluation sub-model, and pronunciation evaluation sub-model are used to perform singing evaluation on the audio to be evaluated, and the corresponding pitch evaluation value, rhythm evaluation value, and pronunciation evaluation value are obtained. Since the multidimensional evaluation model is trained through multidimensional features, and the evaluation values ​​corresponding to each segment of the audio to be evaluated are given based on the three dimensions of pitch, rhythm, and pronunciation, that is, a fine-grained and multi-dimensional evaluation is performed on the audio to be evaluated, the accuracy and interpretability of the singing evaluation are greatly improved.

[0054] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A singing evaluation method based on deep learning, characterized in that: The steps include: Step S10, obtaining a large amount of song data, and cleaning each of the song data; Step S20, extracting the cleaned audio features of each of the singing data to construct a feature data set; Step S30: creating an audio alignment model, and aligning the audio features in the feature data set based on the audio alignment model; Step S40, creating a multidimensional evaluation model based on deep learning, and using the aligned feature data set to train the multidimensional evaluation model, the multidimensional evaluation model includes a pitch evaluation sub-model, a rhythm evaluation sub-model and a pronunciation evaluation sub-model, the input of the pitch evaluation sub-model includes three channels, namely, the reference sample, the high-quality sample, the MFCC, the chromaticity matrix and the beat graph features of the low-quality sample, the input high-dimensional features are mapped to a low-dimensional space through a neural network, and through Triplet Loss and back propagation, the pitch evaluation sub-model is prompted to divide the high-quality samples into a low-dimensional space with a shorter distance, and divide the low-quality samples into a low-dimensional space with a longer distance, and then score each sample based on the distance of the vector in the low-dimensional space; Step S50: obtain the audio to be evaluated, input the audio to be evaluated into a multi-dimensional evaluation model, obtain a pitch evaluation value, a rhythm evaluation value, and a pronunciation evaluation value, and display them.

2. A singing evaluation method based on deep learning as claimed in claim 1, characterized in that: The step S10 is specifically as follows: A large amount of singing data is obtained from different channels, and each of the singing data is sequentially format converted, vocals are separated, blank segments are removed, and noise is reduced.

3. A singing evaluation method based on deep learning as claimed in claim 1, characterized in that: The step S20 is specifically as follows: The cleaned singing data are preprocessed to obtain corresponding frequency domain signals, and audio features including Mel-frequency cepstral coefficients, chromaticity features, beat graphs, signal-to-noise ratio, harmonic noise ratio, frequency perturbations and resonance peaks are extracted based on the frequency domain signals, and a feature data set is constructed based on the audio features.

4. A singing evaluation method based on deep learning as claimed in claim 3, characterized in that: The preprocessing of the cleaned singing data to obtain the corresponding frequency domain signal is specifically: The cleaned singing data are pre-emphasized, framed and windowed in turn to obtain corresponding time domain signals, and each time domain signal is fast Fourier transformed to obtain corresponding frequency domain signals, thereby completing the preprocessing of the singing data.

5. The singing evaluation method based on deep learning as claimed in claim 3, characterized in that: The Mel-frequency cepstrum coefficients are obtained by performing Mel-frequency filter bank filtering, discrete cosine transform, and dynamic difference parameter extraction on the frequency domain signal.

6. A singing evaluation method based on deep learning as claimed in claim 3, characterized in that: The chromaticity features include a chromaticity vector and a chromaticity spectrum.

7. The singing evaluation method based on deep learning as claimed in claim 1, characterized in that: In step S30, the audio alignment model is created based on Guided Attention.

8. The singing evaluation method based on deep learning as claimed in claim 1, characterized in that: The step S50 is specifically as follows: Obtain the audio to be evaluated, clean it, extract audio features and align it in sequence, input it into the pitch evaluation sub-model to obtain a pitch evaluation value, input it into the rhythm evaluation sub-model to obtain a rhythm evaluation value, input it into the pronunciation evaluation sub-model to obtain a pronunciation evaluation value, store and display the pitch evaluation value, rhythm evaluation value and pronunciation evaluation value.

Images