A method and system for evaluating singing level in multiple dimensions

Abstract

The present invention provides a method and system for evaluating singing level in multiple dimensions. Based on collected audio to be evaluated, a pitch prediction model, a singing content recognition model and a multimodal singing emotion recognition model are used in combination to calculate a first score of a singing skill dimension, a second score of a pronunciation clarity dimension and a third score of an emotion dimension. The overall singing level score calculated by combining the first score, the second score and the third score no longer relies on music scores and the reference vocals of the original singer for singing evaluation, but is based on the collected audio to be evaluated to obtain an overall singing level score from a multiple-dimensional evaluation, thereby improving the evaluation effect of the singing level.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and system for evaluating singing level in multiple dimensions. Background Art

[0002] With the popularity of entertainment forms such as karaoke, the assessment of singing level also has broad prospects. Currently, the assessment of singing level mainly relies on music scores and the reference vocals of the original singers.

[0003] However, there are limitations when evaluating singing level based on the music score and the reference vocals of the original singer, which will limit the objectivity and flexibility of the evaluation. When the singer makes adaptations and improvisations during the singing process, it is difficult to accurately evaluate the singing level based only on the music score and the reference vocals of the original singer, and the evaluation effect of the singing level is poor. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method and system for evaluating singing level in multiple dimensions to solve the problem that the evaluation effect of singing level is poor only based on music scores and the reference vocals of the original singer.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of an embodiment of the present invention discloses a method for multi-dimensionally evaluating singing level, the method comprising:

[0007] Collect the audio to be evaluated;

[0008] Inputting the first input data obtained from the audio to be evaluated into a preset pitch prediction model for pitch prediction to obtain a pitch sequence and a hidden layer feature vector sequence, wherein the pitch sequence includes a plurality of pitch prediction values;

[0009] Calculate a first score of the singing skill dimension using the pitch sequence;

[0010] Inputting the second input data obtained from the audio to be evaluated and the hidden layer feature vector sequence into a preset singing content recognition model to perform singing content recognition, thereby obtaining lyrics content and a posterior probability sequence of the lyrics content;

[0011] Calculating a second score of the articulation clarity dimension using the posterior probability sequence of the lyrics content;

[0012] Inputting the third input data obtained from the audio to be evaluated, the hidden layer feature vector sequence and the lyrics content into a preset multimodal singing emotion recognition model to perform singing emotion recognition, and obtaining an emotion posterior probability distribution vector;

[0013] Calculating a third score of the emotion dimension using the emotion posterior probability distribution vector;

[0014] The first score, the second score and the third score are combined to calculate the total score of the singing level;

[0015] Among them, the pitch prediction model, the singing content recognition model and the multimodal singing emotion recognition model are trained based on a sample data set.

[0016] Preferably, the first input data obtained from the audio to be evaluated is input into a preset pitch prediction model for pitch prediction to obtain a pitch sequence and a hidden layer feature vector sequence, including:

[0017] Extracting first input data from the audio to be evaluated, wherein the first input data includes a constant Q transform feature vector sequence and a fundamental frequency sequence;

[0018] The first input data is input into a preset pitch prediction model for pitch prediction to obtain a pitch sequence and a hidden layer feature vector sequence.

[0019] Preferably, using the pitch sequence to calculate a first score of the singing skill dimension includes:

[0020] Using the pitch sequence, calculating one or more of a pitch intonation score, a pitch score, an explosiveness score, a range score, and a subtlety score;

[0021] A first score of the singing skill dimension is obtained by taking a weighted average of one or more of the intonation score, the pitch score, the explosiveness score, the range score and the subtlety score.

[0022] Preferably, the second input data obtained from the audio to be evaluated and the hidden layer feature vector sequence is input into a preset singing content recognition model to perform singing content recognition, and the lyrics content and the posterior probability sequence of the lyrics content are obtained, including:

[0023] Extracting a Mel-spectrogram feature vector sequence of the audio to be evaluated;

[0024] The hidden layer feature vector sequence is merged with the Mel-spectrogram feature vector sequence of the audio to be evaluated to obtain second input data;

[0025] The second input data is input into a preset singing content recognition model to perform singing content recognition, and obtain lyrics content and a posterior probability sequence of the lyrics content.

[0026] Preferably, calculating the second score of the articulation clarity dimension using the posterior probability sequence of the lyrics content includes:

[0027] Determining the pronunciation clarity level of each word in the lyrics content by using the posterior probability of each word in the posterior probability sequence of the lyrics content;

[0028] According to the pronunciation clarity level of each word in the lyrics content, a second score of the pronunciation clarity dimension is calculated.

[0029] Preferably, the third input data obtained from the audio to be evaluated, the hidden layer feature vector sequence and the lyrics content is input into a preset multimodal singing emotion recognition model to perform singing emotion recognition, and obtain an emotion posterior probability distribution vector, including:

[0030] Removing a silent portion from the audio to be evaluated, and dividing the audio to be evaluated from which the silent portion has been removed into a plurality of audio segments;

[0031] Extracting a Mel-spectrogram feature vector sequence of the audio clip;

[0032] Generate third input data corresponding to the audio clip, the third input data including text modality data and audio modality data, the text modality data including the portion of the lyrics content related to the audio clip, and the audio modality data obtained by concatenating the following data: a Mel-spectrogram feature vector sequence of the audio clip and a portion of the hidden layer feature vector sequence related to the audio clip;

[0033] The third input data is input into a preset multimodal singing emotion recognition model to perform singing emotion recognition, and a posterior probability distribution vector of the emotion corresponding to the audio clip is obtained.

[0034] Preferably, calculating the third score of the emotion dimension by using the emotion posterior probability distribution vector includes:

[0035] Filtering out valid clips from all the audio clips according to the maximum value of the posterior probability in the emotion posterior probability distribution vector corresponding to the audio clip;

[0036] Determine the emotion intensity level of the effective segment by using the emotion posterior probability distribution vector corresponding to the effective segment;

[0037] According to the emotion intensity level of the effective segment, a third score of the emotion dimension is calculated.

[0038] A second aspect of an embodiment of the present invention discloses a system for evaluating singing level in multiple dimensions, the system comprising:

[0039] A collection unit, used for collecting audio to be evaluated;

[0040] A first processing unit, configured to input first input data obtained from the audio to be evaluated into a preset pitch prediction model for pitch prediction, to obtain a pitch sequence and a hidden layer feature vector sequence, wherein the pitch sequence includes a plurality of pitch prediction values;

[0041] A first calculation unit, used for calculating a first score of a singing skill dimension by using the pitch sequence;

[0042] A second processing unit is used to input second input data obtained from the audio to be evaluated and the hidden layer feature vector sequence into a preset singing content recognition model to perform singing content recognition, and obtain lyrics content and a posterior probability sequence of the lyrics content;

[0043] A second calculation unit, configured to calculate a second score of the articulation clarity dimension using the posterior probability sequence of the lyrics content;

[0044] A third processing unit is used to input third input data obtained from the audio to be evaluated, the hidden layer feature vector sequence and the lyrics content into a preset multimodal singing emotion recognition model to perform singing emotion recognition, and obtain an emotion posterior probability distribution vector;

[0045] A third calculation unit, configured to calculate a third score of the emotion dimension using the emotion posterior probability distribution vector;

[0046] A fourth calculation unit, configured to calculate a total singing level score by combining the first score, the second score and the third score;

[0047] Among them, the pitch prediction model, the singing content recognition model and the multimodal singing emotion recognition model are trained based on a sample data set.

[0048] Preferably, the first processing unit is specifically used to: extract first input data from the audio to be evaluated, the first input data comprising a constant Q transform feature vector sequence and a fundamental frequency sequence; input the first input data into a preset pitch prediction model for pitch prediction to obtain a pitch sequence and a hidden layer feature vector sequence.

[0049] Preferably, the first calculation unit is specifically used to: use the pitch sequence to calculate one or more of the intonation score, the pitch score, the explosiveness score, the range width score and the delicacy score; take a weighted average of one or more of the intonation score, the pitch score, the explosiveness score, the range width score and the delicacy score to obtain a first score of the singing skill dimension.

[0050] Based on the above-mentioned embodiment of the present invention, a method and system for evaluating singing level in multiple dimensions are provided. On the basis of the collected audio to be evaluated, a pitch prediction model, a singing content recognition model and a multimodal singing emotion recognition model are used in combination to calculate a first score in the singing skill dimension, a second score in the articulation clarity dimension and a third score in the emotion dimension. The first score, the second score and the third score are combined to calculate the total singing level score. The singing evaluation no longer relies on the music score and the reference vocal of the original singer, but is based on the collected audio to be evaluated to obtain the total singing level score from multiple dimensions, thereby improving the evaluation effect of the singing level. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0052] Figure 1 A flowchart of a method for multi-dimensionally evaluating singing level provided by an embodiment of the present invention;

[0053] Figure 2 An example diagram of the principle architecture of a method for multi-dimensionally evaluating singing level provided by an embodiment of the present invention;

[0054] Figure 3 A structural block diagram of a system for multi-dimensionally evaluating singing level provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0057] It can be seen from the background technology that there are limitations when evaluating singing level based on the music score and the reference vocals of the original singer, which will limit the objectivity and flexibility of the evaluation. When the singer adapts and improvises during the singing process, it is difficult to accurately evaluate the singing level based only on the music score and the reference vocals of the original singer, and the evaluation effect of the singing level is poor.

[0058] For example: when a singer covers a published song on stage, he or she will often make a personalized adaptation of the song to show the singer's own creativity and singing style. In this case, it is difficult to accurately evaluate the singer's personalized style and creativity when adapting the song based only on the music score and the reference vocals of the original singer.

[0059] In order to solve the above problems, this solution proposes a method and system for evaluating singing level in multiple dimensions. Based on the collected audio to be evaluated, a pitch prediction model, a singing content recognition model and a multimodal singing emotion recognition model are used to calculate a first score in the singing skill dimension, a second score in the articulation clarity dimension and a third score in the emotion dimension. The first score, the second score and the third score are combined to calculate the total singing level score. The singing evaluation no longer relies on the music score and the reference vocal of the original singer, but is based on the collected audio to be evaluated to obtain the total singing level score from multiple dimensions, so as to improve the evaluation effect of the singing level.

[0060] It is worth noting that, in the subsequent explanation of the embodiments of the present invention, the pitch prediction model, singing content recognition model and multimodal singing emotion recognition model provided by the present invention will be involved, and the pitch prediction model, singing content recognition model and multimodal singing emotion recognition model are trained based on sample data sets; in order to better understand the contents of the subsequent embodiments, the training process of the pitch prediction model, singing content recognition model and multimodal singing emotion recognition model is explained here.

[0061] The sample data set contains multiple sample audios, each of which contains a segment of singing voice, and each of which carries at least corresponding pitch value annotation information (real pitch value), lyrics annotation information (real lyrics content) and emotion annotation information (real emotion).

[0062] Specifically, sample data sets can be collected from the designated platform. In the process of collecting sample data sets, for audio that does not provide vocal tracks, it is necessary to use vocal accompaniment separation and harmony removal technology to extract "pure vocal" audio; the pitch value annotation information includes the duration and pitch value of each note (equivalent to the timestamp and the real pitch value), the lyrics annotation information includes the singing content, and the emotion annotation information includes a variety of real emotions (such as happy, excited, neutral and sad).

[0063] (I) Training process of pitch prediction model:

[0064] Extract the constant-Q transform feature vector sequence and fundamental frequency sequence of each sample audio, wherein the constant-Q transform feature vector sequence is also recorded as CQT (Constant-Q Transform) feature vector sequence;

[0065] The pitch value annotation information is processed to obtain a real pitch sequence and a real boundary sequence, wherein the specific method of obtaining the real boundary sequence is: binarizing the pitch annotation information and converting it into a real boundary sequence composed of real boundary values ​​(with a value of 0 or 1), where 0 represents a silent moment.

[0066] For example: assume that the timestamps and real pitch values ​​contained in the pitch value annotation information are: "0.1-0.2 67", "0.3-0.4 67", "0.5-0.8 68"; after processing the above pitch value annotation information, the real pitch sequence obtained is 67, 67 and 68, and the real boundary sequence obtained (assuming an interval of 0.1) is 1010111.

[0067] The CQT feature vector sequence and fundamental frequency sequence of the sample audio are used as the input of the first model to be trained, and the real pitch sequence and the real boundary sequence are used as the label of the first model to be trained; the first model to be trained adopts a transformer encoder structure, and the first model to be trained is reversely updated with the loss value calculated by the following formulas (1) to (3).

[0068] ;

[0069] ;

[0070] Loss=Loss 1 + α Loss 2 (3);

[0071] From the above formulas (1) to (3), it can be seen that the loss function of the first model to be trained consists of two parts. The first part is the root mean square error Loss of the model predicting the true pitch sequence. 1 The second part is the binary cross entropy loss Loss of the model predicting the true boundary sequence 2 .

[0072] In formulas (1) to (3), L is the length of the true pitch sequence, y i is the i-th true pitch value, is the i-th pitch value predicted by the first model to be trained, wherein the first model to be trained outputs the predicted pitch value at each moment, and the predicted pitch sequence needs to be further masked and filtered according to the real boundary sequence to ensure that the length of the predicted pitch sequence is consistent with the length of the real pitch sequence;

[0073] T1 is the length of the true boundary sequence, b i is the i-th true boundary value (value is 0 or 1), is the i-th boundary value predicted by the first model to be trained; α Loss 2 The first model to be trained continuously reduces the Loss value through the reverse gradient algorithm to optimize the model parameters. When the Loss value converges to the preset range, it indicates that the training of the first model to be trained is completed, and the pitch prediction model used subsequently can be obtained.

[0074] (II) Training process of singing content recognition model:

[0075] The sample audio is input into the trained pitch prediction model to extract the feature vector sequence of the hidden layer in the pitch prediction model (called the sample hidden layer feature vector sequence during the training process). The sample hidden layer feature vector sequence is used to represent the pitch change information. The sample hidden layer feature vector sequence is recorded as H = {h 1 ,h 2 ,…,h J}, h i (i is greater than or equal to 1 and less than or equal to J) is the hidden layer feature vector of the i-th sample, and J is the sequence length.

[0076] Extract the Mel spectrum feature vector sequence of the sample audio (called sample Mel spectrum feature vector sequence during training), and the sample Mel spectrum feature vector sequence is recorded as M={m 1 ,m 2 ,…,m J},m i (i is greater than or equal to 1 and less than or equal to J) is the mel spectrum feature vector of the i-th sample, and the input features of the second model to be trained are fused through the following pitch-aware attention strategy:

[0077] The sample hidden layer feature vector sequence H and the sample Mel spectrum feature vector sequence M are concatenated in the feature dimension to obtain the sample concatenated vector sequence F={f 1 ,f 2 ,…,f J}, f i (i is greater than or equal to 1 and less than or equal to J) is the hidden layer feature vector h of the i-th sample i and the mel spectrum feature vector m of the i-th sample iThe i-th sample concatenation vector is obtained by concatenation; the sample Mel spectrum feature vector sequence M is mapped to the query matrix Q through a fully connected layer, and then the sample concatenation vector sequence F is mapped to the key matrix K and the value matrix V respectively through two fully connected layers, and the attention score matrix A of pitch perception is calculated by formula (4).

[0078] ;

[0079] In formula (4), Represents the transposition operation of the matrix, and the characteristic dimension of each matrix is ​​d h , For the feature dimension d h Square root; attention score matrix A∈R J×J The i-th row and j-th column of represents the attention of the i-th feature vector in the key matrix K to the j-th feature vector of the sample concatenated vector sequence F, R represents a real number, and J represents the dimension of the matrix; by multiplying the attention score matrix A with the value matrix V, the input features of the second model to be trained (that is, the fused feature vector sequence) are obtained; in this way, the pitch changes in singing voices can be processed, so that the second model to be trained can focus more on modeling the feature information of the vocal content during training.

[0080] The second model to be trained for training the singing content recognition model adopts the transformer encoding and decoding structure. The encoder performs a nonlinear transformation on the above-mentioned fused feature vector sequence (the multiplication of the attention score matrix A and the value matrix V), and the decoder further predicts the lyrics content in the form of autoregression. During the training of the second model to be trained, the cross entropy loss function shown in formula (5) is used.

[0081] ;

[0082] In formula (5), c i is the probability distribution vector (the vector dimension is N), c i The probability value of the kth dimension of , S is the sequence length of the posterior probability sequence of the lyrics content predicted by the second model to be trained (equivalent to the number of words in the lyrics); z is the indicator variable, The value of is 0 or 1, where when the true index value of the i-th word in the lyrics content predicted by the second to-be-trained model is k is 1, otherwise, is 0.

[0083] Based on the above content, the second model to be trained is trained to obtain a singing content recognition model for subsequent use.

[0084] (III) Training process of multimodal singing emotion recognition model:

[0085] The above-mentioned trained singing content recognition model is used to extract lyrics from the sample audio, thereby extracting the lyrics content in the sample audio (referred to as sample lyrics content during the training process), and using the sample lyrics content as the text modality input of the third model to be trained; at the same time, the above-mentioned trained pitch prediction model is used to extract the sample hidden layer feature vector sequence corresponding to the sample audio, and the sample hidden layer feature vector sequence and the sample Mel-spectrogram feature vector sequence of the sample audio are spliced ​​in the feature dimension as the audio modality input of the third model to be trained.

[0086] The third model to be trained uses two transformer encoders to model the "text modal input" and "audio modal input" mentioned above, respectively. Finally, the output vectors of the two encoders are weighted fused and mapped into an emotion posterior probability distribution vector through a fully connected layer. The dimension of the emotion posterior probability distribution vector is 4 (for example only, the specific dimension can be set according to actual conditions), corresponding to the prediction probabilities of the four emotions respectively. The cross entropy loss function is used in the training process of the third model to be trained.

[0087] Based on the above content, the third model to be trained is trained to obtain a multimodal singing emotion recognition model for subsequent use.

[0088] It is worth noting that during the training process of the third model to be trained, the input of the text modality is the real lyrics content, while during inference, the trained singing content recognition model is needed to recognize the lyrics of the sample audio. Since there may be an error probability in lyrics recognition, the proportion of the text modality output vector will be appropriately reduced during weighted fusion, thereby stabilizing the prediction ability of the third model to be trained.

[0089] The contents of (i) to (iii) above are related explanations of the training process of the pitch prediction model, singing content recognition model and multimodal singing emotion recognition model provided by the present invention.

[0090] See also Figure 1 , showing a flow chart of a method for multi-dimensionally evaluating singing level provided by an embodiment of the present invention, the method comprising:

[0091] Step S101: collecting audio to be evaluated.

[0092] In the specific implementation of step S101, the audio of the singer during the singing of the song is collected to obtain the audio to be evaluated.

[0093] Step S102: inputting the first input data obtained from the audio to be evaluated into a preset pitch prediction model for pitch prediction to obtain a pitch sequence and a hidden layer feature vector sequence.

[0094] It should be noted that the training process of the pitch prediction model has been explained in advance and will not be repeated here.

[0095] In the specific implementation of step S102, first input data is extracted from the audio to be evaluated, and the first input data includes a constant Q transform feature vector sequence and a fundamental frequency sequence, that is, the first input data includes a constant Q transform feature vector sequence and a fundamental frequency sequence extracted from the audio to be evaluated.

[0096] Inputting the first input data into a pitch prediction model to perform pitch prediction, thereby obtaining a pitch sequence and a hidden layer feature vector sequence;

[0097] That is, the first input data is input into a pitch prediction model, a pitch sequence corresponding to the audio to be evaluated output by the pitch prediction model is obtained, and a hidden layer feature vector sequence corresponding to the audio to be evaluated is extracted from the pitch prediction model.

[0098] The pitch sequence predicted by the pitch prediction model contains multiple pitch prediction values ​​(also called predicted pitch values), and the pitch sequence is denoted as S={s 1 ,s 2 ,…,s L}, where L is the length of the pitch sequence, s 1 to L Predicted values ​​for each pitch.

[0099] Step S103: Calculate the first score of the singing skill dimension using the pitch sequence.

[0100] In the process of implementing step S103, after obtaining the pitch sequence corresponding to the audio to be evaluated predicted by the pitch prediction model, the singing dimension is evaluated from aspects such as intonation, pitch, explosive power, range and delicacy using each pitch prediction value in the pitch sequence, so as to calculate the first score of the singing dimension.

[0101] In some embodiments, the specific implementation method of using the pitch sequence to calculate the first score of the singing skill dimension is: using the pitch sequence to calculate one or more of the intonation score, pitch score, explosiveness score, range width score and delicacy score; taking a weighted average of one or more of the intonation score, pitch score, explosiveness score, range width score and delicacy score to obtain the first score of the singing skill dimension.

[0102] Specifically, any one or more of the intonation score, pitch score, explosiveness score, range score, and subtlety score are calculated, and then the score calculation results are weighted averaged to obtain the first score of the singing skill dimension.

[0103] For example: If you calculate the intonation score, pitch score, explosiveness score, range score and delicacy score, then take a weighted average of the intonation score, pitch score, explosiveness score, range score and delicacy score to get the first score of the singing skill dimension.

[0104] For another example: If the intonation score, pitch score and delicacy score are calculated, the intonation score, pitch score and delicacy score are weighted averaged to obtain the first score of the singing skill dimension.

[0105] The following explains how to calculate the intonation score, pitch score, explosiveness score, range score and subtlety score.

[0106] (1) Method for calculating intonation score:

[0107] It should be noted that the pitch intonation score mainly focuses on the accuracy of the pitch value. The present invention sets the loss function of the pitch prediction model so that the pitch prediction value in the pitch sequence output by the pitch prediction model is a floating point value accurate to decimal places.

[0108] Since the pitch standard value (also called the standard pitch value) is an integer value, the present invention first determines the pitch standard value closest to each pitch prediction value; for each pitch prediction value, the pitch accuracy level of the pitch prediction value is determined by the error value between "the pitch prediction value" and "the pitch standard value closest to the pitch prediction value".

[0109] Specifically, for each pitch prediction value in the pitch sequence, first determine the pitch standard value closest to the pitch prediction value, and then calculate the error value between the pitch prediction value and the pitch standard value closest to it. The error value can effectively measure the accuracy of the pitch, and the pitch accuracy level of the pitch prediction value is determined according to the error value and the following formula (6).

[0110] ;

[0111] In formula (6), level a is the pitch prediction value’s pitch accuracy level, e represents the error between a “pitch prediction value” and “the closest pitch standard value”, and the pitch accuracy level is divided into l 1 , l 2 and l 3 Three levels, l1 Express excellence, l 2 It means good, l 3 Indicates qualified.

[0112] After determining the pitch level of each pitch prediction value in the pitch sequence, the pitch score Score is calculated using the pitch level of each pitch prediction value and formula (7). a .

[0113] ;

[0114] In formula (7), n i To be rated l i The number of pitch predictions for this pitch level, L is the length of the pitch sequence, satisfies n 1 +n 2 +n 3 =L; if all pitch prediction values ​​have an excellent intonation rating (all pitch prediction values ​​have an excellent intonation rating l 1 ), then the pitch score Score a The full score is (100 points).

[0115] (2) Calculate the pitch score:

[0116] The pitch score (pitch score) mainly considers the singer's ability to sing high notes, so the pitch prediction value is divided into pitch levels using formula (8).

[0117] ;

[0118] In formula (8), level b is the pitch level of the predicted pitch value, s is a certain predicted pitch value, the pitch value 81 and the pitch value 69 correspond to the scale A5 and the scale A4 respectively (these two scales belong to the high frequency band), and the pitch level is divided into l 4 , l 5 and l 6 Three levels, l 4 Express excellence, l 5 It means good, l 6 Indicates qualified.

[0119] After the pitch level of each pitch prediction value is determined, the pitch score can be calculated based on the pitch level of the pitch prediction value by referring to the method of calculating the intonation score in the above formula (7).

[0120] (3) Method for calculating explosive power score:

[0121] The explosive power score mainly considers the singer's instantaneous pitch transition ability. In the process of calculating the explosive power score, the pitch sequence is firstly differentiated to obtain the pitch difference sequence, and then the explosive power level is divided for some sequences in the pitch difference sequence with positive difference values.

[0122] Specifically, after obtaining the pitch difference sequence, the explosiveness level is divided according to formula (9) for the partial sequence with positive difference values.

[0123] ;

[0124] In formula (9), level c is the explosive power level, s delta is the pitch difference between a certain pitch prediction value and its previous pitch prediction value. The explosiveness level is divided into l 7 , l 8 and l 9 Three levels, l 7 Express excellence, l 8 It means good, l 9 Indicates qualified.

[0125] After completing the classification of explosive power levels, the method of calculating the intonation score can be referred to the above formula (7) to calculate the explosive power score based on the classification results of the explosive power levels.

[0126] (4) Method for calculating the range score:

[0127] The scoring of vocal range breadth focuses on considering the breadth of the singer's vocal range. By calculating the difference between the largest pitch prediction value and the smallest pitch prediction value in the pitch sequence, the singer's vocal range breadth can be obtained; then the vocal range breadth grade to which the singer's vocal range breadth belongs is determined; finally, the score corresponding to the vocal range breadth grade to which the singer's vocal range breadth belongs is determined to obtain the vocal range breadth score.

[0128] (5) Method for calculating the fineness score:

[0129] The scoring of subtlety mainly focuses on the diversity of the singer's pitch when singing. In singing, the richness of pitch types directly reflects the singer's delicate interpretation ability.

[0130] In the process of calculating the singer's delicacy score, the pitch sequence is rounded to obtain an integer pitch sequence; then, the integer pitch sequence is deduplicated, and the number of pitch types in the deduplicated integer pitch sequence is counted;

[0131] After obtaining the number of pitch types through statistics, the quantity level to which the number of pitch types belongs is determined; finally, the score corresponding to the quantity level to which the number of pitch types belongs is determined to obtain a delicacy score.

[0132] The above are instructions on how to calculate the intonation score, pitch score, explosiveness score, range score, and subtlety score.

[0133] After calculating the intonation score, pitch score, explosiveness score, range score and delicacy score in the above manner, the calculated intonation score, pitch score, explosiveness score, range score and delicacy score are weighted averaged to calculate the singer's first score in the singing skill dimension.

[0134] Step S104: inputting the second input data obtained from the audio to be evaluated and the hidden layer feature vector sequence into a preset singing content recognition model to perform singing content recognition, and obtaining the lyrics content and the posterior probability sequence of the lyrics content.

[0135] It should be noted that the training process of the singing content recognition model has been explained in advance and will not be repeated here.

[0136] In the specific implementation of step S104, the mel-spectrogram feature vector sequence of the audio to be evaluated is extracted; the hidden layer feature vector sequence and the mel-spectrogram feature vector sequence of the audio to be evaluated are fused to obtain second input data; the second input data is input into the singing content recognition model to perform singing content recognition to obtain the lyrics content and the posterior probability sequence of the lyrics content.

[0137] That is, extracting a Mel-spectrogram feature vector sequence of the audio to be evaluated, and fusing the Mel-spectrogram feature vector sequence with a hidden layer feature vector sequence corresponding to the audio to be evaluated obtained from the pitch prediction model to obtain second input data;

[0138] The second input data is input into the singing content recognition model to obtain the posterior probability sequence of the lyrics content output by the singing content recognition model, and the lyrics content is obtained through a decoding strategy.

[0139] Among them, the posterior probability sequence P of the lyrics content is P={p 1 ,p 1 ,…,p S}, where S represents the length of the posterior probability sequence.

[0140] Step S105: Calculate a second score of the articulation clarity dimension using the posterior probability sequence of the lyrics content.

[0141] In the specific implementation of step S105, the posterior probability of each word in the posterior probability sequence of the lyrics content is used to determine the pronunciation clarity level of each word in the lyrics content; according to the pronunciation clarity level of each word in the lyrics content, the second score of the pronunciation clarity dimension is calculated.

[0142] Specifically, the posterior probability of each word in the posterior probability sequence of the lyrics content is used to divide the pronunciation clarity level through formula (10).

[0143] ;

[0144] In formula (10), level d is the articulation clarity level, p is the posterior probability of a word in the posterior probability sequence of the lyrics content, and the articulation clarity level is divided into l 10 , l 11 and l 12 Three levels, l 10 Express excellence, l 11 It means good, l 12 Indicates qualified.

[0145] After completing the division of articulation clarity levels, the method of calculating the intonation score can refer to the above formula (7) to calculate the second score of the articulation clarity dimension based on the articulation clarity level of each word in the lyrics content.

[0146] Step S106: inputting the third input data obtained from the audio to be evaluated, the hidden layer feature vector sequence and the lyrics content into a preset multimodal singing emotion recognition model to perform singing emotion recognition, and obtain an emotion posterior probability distribution vector.

[0147] It should be noted that the training process of the multimodal singing emotion recognition model has been explained in advance and will not be repeated here.

[0148] In the specific implementation of step S106, the silent part in the audio to be evaluated is removed, and the audio to be evaluated with the silent part removed is divided into multiple audio segments; the Mel spectrum feature vector sequence of the audio segment is extracted; and the third input data corresponding to the audio segment is generated, the third input data includes text modality data and audio modality data, the text modality data includes the part of the lyrics content related to the audio segment, and the audio modality data is obtained by splicing the following data: the Mel spectrum feature vector sequence of the audio segment and the part of the hidden layer feature vector sequence related to the audio segment;

[0149] The third input data is input into the multimodal singing emotion recognition model to perform singing emotion recognition, and obtain the emotion posterior probability distribution vector corresponding to the audio clip.

[0150] Specifically, for the hidden layer feature vector sequence and lyrics content corresponding to the audio to be evaluated obtained from the pitch prediction model and the singing content recognition model, the Montreal forced alignment algorithm is used to calculate the timestamp corresponding to each word in the lyrics content;

[0151] Performing silence detection on the audio to be evaluated, discarding the silent part in the audio to be evaluated, and dividing the audio to be evaluated with the silent part discarded into multiple audio segments (equivalent to the non-silent parts);

[0152] For each audio clip, based on the timestamp information of the audio clip, from the hidden layer feature vector sequence and the lyrics content corresponding to the entire audio to be evaluated, locate the part of the hidden layer feature vector sequence related to the audio clip, and locate the part of the lyrics content related to the audio clip;

[0153] That is, the part of the hidden layer feature vector sequence related to each audio clip is determined, and the part of the lyrics content related to each audio clip is determined.

[0154] For each audio clip, generate the third input data corresponding to the audio clip, the third input data includes text modal data and audio modal data, the text modal data = the part of the lyrics content related to the audio clip, the audio modal data = the mel-spectrogram feature vector sequence of the audio clip + the part of the hidden layer feature vector sequence related to the audio clip.

[0155] For each audio clip, the third input data corresponding to the audio clip is input into the multimodal singing emotion recognition model to perform singing emotion recognition, so as to obtain the emotion posterior probability distribution vector corresponding to the audio clip.

[0156] Step S107: Calculate the third score of the emotion dimension using the emotion posterior probability distribution vector.

[0157] It should be noted that the emotion posterior probability distribution vector corresponding to the audio clip contains the posterior probabilities of multiple emotions predicted by the multimodal singing emotion recognition model, that is, the multiple dimensions of the emotion posterior probability distribution vector represent the posterior probabilities of multiple emotions respectively;

[0158] For example: "happy" and "excited" are classified as positive emotions, and "sad" is classified as negative emotion; the four dimensions of the emotion posterior probability distribution vector corresponding to the audio clip represent the posterior probabilities of the four emotions "happy", "excited", "neutral" and "sad" respectively; the posterior probabilities of the four emotions "happy", "excited", "neutral" and "sad" in the emotion posterior probability distribution vector corresponding to the audio clip are recorded as g1, g2, g3 and g4 respectively, g1+g2+g3+g4=1.

[0159] In the specific implementation of step S107, after obtaining the emotion posterior probability distribution vector corresponding to each audio segment, the valid segment (valid audio segment) is screened out from all audio segments according to the maximum value of the posterior probability in the emotion posterior probability distribution vector corresponding to the audio segment;

[0160] The emotion posterior probability distribution vector corresponding to the effective segment is used to determine the emotion intensity level of the effective segment; and the third score of the emotion dimension is calculated according to the emotion intensity level of the effective segment.

[0161] Specifically, for a certain audio clip, if the maximum value of the posterior probability in the emotion posterior probability distribution vector corresponding to the audio clip is less than the probability threshold (for example, less than 0.5), the audio clip is characterized as an invalid clip; if the maximum value of the posterior probability in the emotion posterior probability distribution vector corresponding to the audio clip is greater than or equal to the probability threshold, the audio clip is characterized as a valid clip;

[0162] The audio clips are screened in the above manner, invalid clips (audio clips whose maximum posterior probability is less than the probability threshold) are screened out and discarded, and valid clips (audio clips whose maximum posterior probability is greater than or equal to the probability threshold) are retained.

[0163] For each effective segment obtained by screening, the emotion posterior probability distribution vector corresponding to the effective segment is used to determine the emotion intensity value Emo of the effective segment through formula (11).

[0164] Emo=|g1+g2-g4|(11);

[0165] In formula (11), g1 is the posterior probability of the emotion “happy” in the emotion posterior probability distribution vector, g2 is the posterior probability of the emotion “excited” in the emotion posterior probability distribution vector, and g4 is the posterior probability of the emotion “sad” in the emotion posterior probability distribution vector.

[0166] After calculating the emotion intensity value of each valid segment, for each valid segment, the emotion intensity level of the valid segment is determined using the emotion intensity value of the valid segment and formula (12).

[0167] ;

[0168] In formula (12), level e is the emotional intensity level, Emo is the emotional intensity value, and the emotional intensity level is divided into l 13 , l 14 and l 15 Three levels, l 13 Express excellence, l 14 It means good, l 15 Indicates qualified.

[0169] After determining the emotional intensity level of each valid segment, the third score of the emotional dimension can be calculated based on the emotional intensity level of the valid segment by referring to the method of calculating the pitch score in the above formula (7).

[0170] Step S108: Calculate the total singing level score by combining the first score, the second score and the third score.

[0171] In the specific implementation process of step S108, after obtaining a first score for the singing skill dimension, a second score for the articulation clarity dimension, and a third score for the emotion dimension based on the audio to be evaluated, the first score, the second score, and the third score are weightedly fused to obtain an overall score for the singer's singing level.

[0172] Specifically, the first score, the second score and the third score are weightedly fused using the weights corresponding to the singing skills dimension, the pronunciation clarity dimension and the emotional dimension to calculate the total singing level score, that is, the total singing level score = a*first score + b*second score + c*third score, where a, b and c are weights, a+b+c=1.

[0173] It should be noted that the weights corresponding to the singing skills dimension, pronunciation clarity dimension and emotional dimension can be set according to actual conditions.

[0174] In actual applications, one of the application scenarios of the present invention is: in a singing competition scenario, the vocal sub-track data of the singer (audio to be evaluated) is obtained, and the singer's total singing level score is calculated based on the singer's vocal sub-track data; the singer's total singing level score is used to predict the singing ranking; among them, the weights corresponding to the singing skills dimension, the pronunciation clarity dimension and the emotion dimension can be adjusted and set by collecting data and competition results of multiple historical singing competitions.

[0175] In an embodiment of the present invention, based on the collected audio to be evaluated, a pitch prediction model, a singing content recognition model and a multimodal singing emotion recognition model are used in combination to calculate a first score for the singing skill dimension, a second score for the articulation clarity dimension and a third score for the emotion dimension. The first score, the second score and the third score are combined to calculate the total singing level score. The singing evaluation no longer relies on the music score and the reference vocal of the original singer, but is based on the collected audio to be evaluated to obtain the total singing level score from a multi-dimensional evaluation, thereby improving the evaluation effect of the singing level.

[0176] In the above-mentioned embodiments of the present invention Figure 1 Based on this, it can be seen that the principle structure diagram of a method for multi-dimensionally evaluating singing level provided by the present invention is as follows Figure 2 shown.

[0177] Figure 2 In the method, the audio to be evaluated is processed by the pitch prediction model to obtain the hidden layer feature vector sequence corresponding to the audio to be evaluated, and the first score of the singing skill dimension is obtained; wherein, the first score of the singing skill dimension is obtained by weighted averaging the intonation score, pitch score, explosiveness score, range score and delicacy score.

[0178] By processing the audio to be evaluated and its corresponding hidden layer feature vector sequence through the singing content recognition model, a second score in the dimensions of lyrics content and pronunciation clarity can be obtained.

[0179] By processing the audio to be evaluated, the lyrics content and the hidden layer feature vector sequence through the multimodal singing emotion recognition model, the third score of the emotion dimension can be obtained.

[0180] The first score, the second score and the third score obtained above are weightedly integrated to obtain the singer's overall singing level score.

[0181] Corresponding to the method for evaluating singing level in multiple dimensions provided by the above embodiment of the present invention, see Figure 3, the embodiment of the present invention also provides a structural block diagram of a system for multi-dimensionally evaluating singing level, the system comprising: a collection unit 301, a first processing unit 302, a first calculation unit 303, a second processing unit 304, a second calculation unit 305, a third processing unit 306, a third calculation unit 307 and a fourth calculation unit 308;

[0182] The collection unit 301 is used to collect audio to be evaluated.

[0183] The first processing unit 302 is used to input the first input data obtained from the audio to be evaluated into a preset pitch prediction model for pitch prediction to obtain a pitch sequence and a hidden layer feature vector sequence, wherein the pitch sequence includes multiple pitch prediction values.

[0184] In a specific implementation, the first processing unit 302 is specifically used to: extract first input data from the audio to be evaluated, the first input data comprising a constant Q transform feature vector sequence and a fundamental frequency sequence; input the first input data into a preset pitch prediction model for pitch prediction to obtain a pitch sequence and a hidden layer feature vector sequence.

[0185] The first calculation unit 303 is used to calculate a first score of the singing skill dimension by using the pitch sequence.

[0186] In the specific implementation, the first calculation unit 303 is specifically used to: use the pitch sequence to calculate one or more of the intonation score, pitch score, explosiveness score, range width score and delicateness score; take a weighted average of one or more of the intonation score, pitch score, explosiveness score, range width score and delicateness score to obtain a first score of the singing skill dimension.

[0187] The second processing unit 304 is used to input the second input data obtained from the audio to be evaluated and the hidden layer feature vector sequence into a preset singing content recognition model to perform singing content recognition, and obtain the lyrics content and the posterior probability sequence of the lyrics content.

[0188] The second calculation unit 305 is used to calculate a second score of the articulation clarity dimension by using the posterior probability sequence of the lyrics content.

[0189] In a specific implementation, the second calculation unit 305 is specifically used to: determine the pronunciation clarity level of each word in the lyrics content using the posterior probability of each word in the posterior probability sequence of the lyrics content; and calculate the second score of the pronunciation clarity dimension according to the pronunciation clarity level of each word in the lyrics content.

[0190] The third processing unit 306 is used to input the third input data obtained from the audio to be evaluated, the hidden layer feature vector sequence and the lyrics content into a preset multimodal singing emotion recognition model to perform singing emotion recognition and obtain an emotion posterior probability distribution vector.

[0191] The third calculation unit 307 is used to calculate the third score of the emotion dimension by using the emotion posterior probability distribution vector.

[0192] The fourth calculation unit 308 is used to calculate the total singing level score by combining the first score, the second score and the third score;

[0193] Among them, the pitch prediction model, singing content recognition model and multimodal singing emotion recognition model are trained based on sample data sets.

[0194] In an embodiment of the present invention, based on the collected audio to be evaluated, a pitch prediction model, a singing content recognition model and a multimodal singing emotion recognition model are used in combination to calculate a first score for the singing skill dimension, a second score for the articulation clarity dimension and a third score for the emotion dimension. The first score, the second score and the third score are combined to calculate the total singing level score. The singing evaluation no longer relies on the music score and the reference vocal of the original singer, but is based on the collected audio to be evaluated to obtain the total singing level score from a multi-dimensional evaluation, thereby improving the evaluation effect of the singing level.

[0195] Preferably, combined Figure 3 The second processing unit 304 includes an extraction module, a fusion module and a content identification module. The execution principle of each module is as follows:

[0196] The extraction module is used to extract the Mel spectrum feature vector sequence of the audio to be evaluated.

[0197] The fusion module is used to fuse the hidden layer feature vector sequence with the Mel spectrum feature vector sequence of the audio to be evaluated to obtain the second input data.

[0198] The content recognition module is used to input the second input data into a preset singing content recognition model to perform singing content recognition, and obtain the lyrics content and the posterior probability sequence of the lyrics content.

[0199] Preferably, combined Figure 3 The third processing unit 306 includes a segmentation module, an extraction module, a generation module and an emotion recognition module. The execution principle of each module is as follows:

[0200] The segmentation module is used to remove the silent part in the audio to be evaluated, and segment the audio to be evaluated with the silent part removed into multiple audio segments.

[0201] The extraction module is used to extract the Mel spectrum feature vector sequence of the audio clip.

[0202] A generation module is used to generate third input data corresponding to the audio clip, the third input data includes text modal data and audio modal data, the text modal data contains the part of the lyrics content related to the audio clip, and the audio modal data is obtained by splicing the following data: the Mel spectrum feature vector sequence of the audio clip, and the part of the hidden layer feature vector sequence related to the audio clip.

[0203] The emotion recognition module is used to input the third input data into a preset multimodal singing emotion recognition model to perform singing emotion recognition and obtain an emotion posterior probability distribution vector corresponding to the audio clip.

[0204] Correspondingly, the third calculation unit 307 is specifically used to: screen out valid segments from all audio segments according to the maximum value of the posterior probability in the emotion posterior probability distribution vector corresponding to the audio segment; determine the emotion intensity level of the valid segment using the emotion posterior probability distribution vector corresponding to the valid segment; and calculate the third score of the emotion dimension according to the emotion intensity level of the valid segment.

[0205] In summary, the embodiments of the present invention provide a method and system for evaluating singing level in multiple dimensions. Based on the collected audio to be evaluated, a pitch prediction model, a singing content recognition model and a multimodal singing emotion recognition model are used to calculate a first score in the singing skill dimension, a second score in the articulation clarity dimension and a third score in the emotion dimension. The first score, the second score and the third score are combined to calculate the total singing level score. The singing evaluation no longer relies on the music score and the reference vocal of the original singer, but is based on the collected audio to be evaluated to obtain the total singing level score from multiple dimensions, thereby improving the evaluation effect of the singing level.

[0206] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0207] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0208] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for evaluating singing level in multiple dimensions, characterized in that: The method comprises: Collect the audio to be evaluated; Inputting the first input data obtained from the audio to be evaluated into a preset pitch prediction model for pitch prediction to obtain a pitch sequence and a hidden layer feature vector sequence, wherein the pitch sequence includes a plurality of pitch prediction values; Calculate a first score of the singing skill dimension using the pitch sequence; Inputting the second input data obtained from the audio to be evaluated and the hidden layer feature vector sequence into a preset singing content recognition model to perform singing content recognition, thereby obtaining lyrics content and a posterior probability sequence of the lyrics content; Calculating a second score of the articulation clarity dimension using the posterior probability sequence of the lyrics content; Removing a silent portion from the audio to be evaluated, and dividing the audio to be evaluated from which the silent portion has been removed into a plurality of audio segments; Extracting a Mel-spectrogram feature vector sequence of the audio clip; Generate third input data corresponding to the audio clip, the third input data including text modality data and audio modality data, the text modality data including the portion of the lyrics content related to the audio clip, and the audio modality data obtained by concatenating the following data: a Mel-spectrogram feature vector sequence of the audio clip and a portion of the hidden layer feature vector sequence related to the audio clip; Inputting the third input data into a preset multimodal singing emotion recognition model to perform singing emotion recognition, and obtaining an emotion posterior probability distribution vector corresponding to the audio clip; Calculating a third score of the emotion dimension using the emotion posterior probability distribution vector; The first score, the second score and the third score are combined to calculate the total score of the singing level; Among them, the pitch prediction model, the singing content recognition model and the multimodal singing emotion recognition model are trained based on a sample data set.

2. The method according to claim 1, characterized in that Inputting the first input data obtained from the audio to be evaluated into a preset pitch prediction model for pitch prediction to obtain a pitch sequence and a hidden layer feature vector sequence, including: Extracting first input data from the audio to be evaluated, wherein the first input data includes a constant Q transform feature vector sequence and a fundamental frequency sequence; The first input data is input into a preset pitch prediction model for pitch prediction to obtain a pitch sequence and a hidden layer feature vector sequence.

3. The method according to claim 1 or 2, characterized in that: Using the pitch sequence, calculating a first score of the singing skill dimension includes: Using the pitch sequence, calculating one or more of a pitch intonation score, a pitch score, an explosiveness score, a range score, and a subtlety score; A first score of the singing skill dimension is obtained by taking a weighted average of one or more of the intonation score, the pitch score, the explosiveness score, the range score and the subtlety score.

4. The method according to claim 1, characterized in that: Inputting the second input data obtained from the audio to be evaluated and the hidden layer feature vector sequence into a preset singing content recognition model to perform singing content recognition, and obtaining lyrics content and a posterior probability sequence of the lyrics content, including: Extracting a Mel-spectrogram feature vector sequence of the audio to be evaluated; The hidden layer feature vector sequence is merged with the Mel-spectrogram feature vector sequence of the audio to be evaluated to obtain second input data; The second input data is input into a preset singing content recognition model to perform singing content recognition, and obtain lyrics content and a posterior probability sequence of the lyrics content.

5. The method according to claim 1 or 4, characterized in that: Calculating a second score of the articulation clarity dimension using the posterior probability sequence of the lyrics content includes: Determining the pronunciation clarity level of each word in the lyrics content by using the posterior probability of each word in the posterior probability sequence of the lyrics content; According to the pronunciation clarity level of each word in the lyrics content, a second score of the pronunciation clarity dimension is calculated.

6. The method according to claim 1, characterized in that Calculating a third score of the emotion dimension using the emotion posterior probability distribution vector includes: Filtering out valid clips from all the audio clips according to the maximum value of the posterior probability in the emotion posterior probability distribution vector corresponding to the audio clip; Determine the emotion intensity level of the effective segment by using the emotion posterior probability distribution vector corresponding to the effective segment; According to the emotion intensity level of the effective segment, a third score of the emotion dimension is calculated.

7. A system for evaluating singing level in multiple dimensions, characterized in that: The system comprises: A collection unit, used for collecting audio to be evaluated; A first processing unit, configured to input first input data obtained from the audio to be evaluated into a preset pitch prediction model for pitch prediction, to obtain a pitch sequence and a hidden layer feature vector sequence, wherein the pitch sequence includes a plurality of pitch prediction values; A first calculation unit, used for calculating a first score of a singing skill dimension by using the pitch sequence; A second processing unit is used to input second input data obtained from the audio to be evaluated and the hidden layer feature vector sequence into a preset singing content recognition model to perform singing content recognition, and obtain lyrics content and a posterior probability sequence of the lyrics content; A second calculation unit, configured to calculate a second score of the articulation clarity dimension using the posterior probability sequence of the lyrics content; A third processing unit is used to input third input data obtained from the audio to be evaluated, the hidden layer feature vector sequence and the lyrics content into a preset multimodal singing emotion recognition model to perform singing emotion recognition, and obtain an emotion posterior probability distribution vector; A third calculation unit, configured to calculate a third score of the emotion dimension using the emotion posterior probability distribution vector; A fourth calculation unit, configured to calculate a total singing level score by combining the first score, the second score and the third score; Wherein, the pitch prediction model, the singing content recognition model and the multimodal singing emotion recognition model are trained based on a sample data set; The third processing unit includes a segmentation module, an extraction module, a generation module and an emotion recognition module; The segmentation module is used to remove the silent part in the audio to be evaluated, and segment the audio to be evaluated with the silent part removed into multiple audio segments; The extraction module is used to extract the Mel spectrum feature vector sequence of the audio clip; The generating module is used to generate third input data corresponding to the audio clip, the third input data includes text modality data and audio modality data, the text modality data includes the part of the lyrics content related to the audio clip, and the audio modality data is obtained by concatenating the following data: the mel spectrum feature vector sequence of the audio clip and the part of the hidden layer feature vector sequence related to the audio clip; The emotion recognition module is used to input the third input data into a preset multimodal singing emotion recognition model to perform singing emotion recognition, and obtain an emotion posterior probability distribution vector corresponding to the audio clip.

8. The system according to claim 7, characterized in that The first processing unit is specifically used to: extract first input data from the audio to be evaluated, the first input data comprising a constant Q transform feature vector sequence and a fundamental frequency sequence; input the first input data into a preset pitch prediction model for pitch prediction to obtain a pitch sequence and a hidden layer feature vector sequence.

9. The system according to claim 7 or 8, characterized in that: The first calculation unit is specifically used to: use the pitch sequence to calculate one or more of the intonation score, the pitch score, the explosiveness score, the range width score and the delicacy score; take a weighted average of one or more of the intonation score, the pitch score, the explosiveness score, the range width score and the delicacy score to obtain a first score of the singing skill dimension.

Images