Singing scoring methods and related devices, equipment and storage media

Abstract

This application discloses a singing scoring method and related apparatus, equipment, and storage medium. The singing scoring method includes: acquiring the singing audio to be evaluated and the target score of the target song to which the singing audio belongs; extracting the acoustic features of the singing audio and extracting the singing sequence of the target score; wherein the singing sequence includes the pitch and beat length of the target song at each singing moment; performing feature extraction on the acoustic features based on a first extraction model to obtain a first feature representation, and performing feature extraction on the singing sequence based on a second extraction model to obtain a second feature representation; wherein both the first and second extraction models adopt a target network framework, which includes a sequentially connected convolutional neural network and a recurrent neural network; and making a prediction based on the first and second feature representations to obtain the singing score of the singing audio. The above scheme can improve the applicability, accuracy, robustness, and generalization ability of singing scoring.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to a singing scoring method and related apparatus, devices and storage media. Background Technology

[0002] Currently, learning music through online platforms and smart devices is receiving increasing attention. Traditional singing evaluation mainly relies on manual scoring, which is not only time-consuming and labor-intensive but also easily influenced by subjective factors.

[0003] To overcome the shortcomings of manual scoring, automated scoring methods such as rule-based scoring and machine learning-based scoring have gradually emerged. However, the former lacks flexibility and struggles to adapt to the performance differences of various singers, while the latter relies on manually designed scoring features, lacking robustness and generalization ability. Therefore, improving the applicability, accuracy, robustness, and generalization ability of singing scoring has become an urgent problem to be solved. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a singing scoring method and related apparatus, equipment, and storage medium, which can improve the applicability, accuracy, robustness, and generalization ability of singing scoring.

[0005] To address the aforementioned technical problems, the first aspect of this application provides a singing scoring method, comprising: acquiring a singing audio to be evaluated and a target score of a target song to which the singing audio belongs; extracting acoustic features of the singing audio and extracting a performance sequence of the target score; wherein the performance sequence includes the pitch and beat length of the target song at each performance moment; performing feature extraction on the acoustic features based on a first extraction model to obtain a first feature representation, and performing feature extraction on the performance sequence based on a second extraction model to obtain a second feature representation; wherein both the first and second extraction models employ a target network framework, the target network framework including a sequentially connected convolutional neural network and a recurrent neural network; and making a prediction based on the first and second feature representations to obtain a performance score for the singing audio.

[0006] To address the aforementioned technical problems, a second aspect of this application provides a singing scoring device, comprising: an acquisition and preparation module, a content extraction module, a feature extraction module, and a score prediction module. The acquisition and preparation module is used to acquire the singing audio to be evaluated and the target score of the target song to which the singing audio belongs. The content extraction module is used to extract the acoustic features of the singing audio and extract the singing sequence of the target score; wherein the singing sequence includes the pitch and beat length of the target song at each singing moment in sequence. The feature extraction module is used to perform feature extraction on the acoustic features based on a first extraction model to obtain a first feature representation, and to perform feature extraction on the singing sequence based on a second extraction model to obtain a second feature representation; wherein both the first extraction model and the second extraction model adopt a target network framework, the target network framework including a sequentially connected convolutional neural network and a recurrent neural network. The score prediction module is used to predict the singing score of the singing audio based on the first feature representation and the second feature representation.

[0007] To address the aforementioned technical problems, a third aspect of this application provides an electronic device comprising at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor executes the program instructions to implement the singing scoring method of the first aspect.

[0008] To address the aforementioned technical problems, a fourth aspect of this application provides a computer-readable storage medium storing program instructions executable by a processor, the program instructions being used to implement the singing scoring method of the first aspect described above.

[0009] The above scheme obtains the singing audio to be evaluated and the target score of the target song to which the singing audio belongs. Then, it extracts the acoustic features of the singing audio and the singing sequence of the target score, including the pitch and beat length of the target song at each moment of performance. Based on a first extraction model, it extracts features from the acoustic features to obtain a first feature representation. Then, based on a second extraction model, it extracts features from the singing sequence to obtain a second feature representation. Both the first and second extraction models use a target network framework, which includes sequentially connected convolutional neural networks and recurrent neural networks. Finally, it predicts the singing score based on the first and second feature representations. This approach directly extracts the corresponding feature representations from the singing audio and the target score. This method of score prediction, compared to rule-based scoring, is more adaptable to the performance differences of various singers, thus improving the applicability of singing scores. Furthermore, because the score prediction process references not only the primary feature representation extracted from the singing audio but also the target score of the corresponding song, using the singing sequence to aid prediction, it helps improve the accuracy of singing scores. Additionally, the feature extraction process combines convolutional neural networks and recurrent neural networks. The former focuses on local features such as notes, melody, and pitch changes, while the latter captures temporal dynamic features such as rhythm changes and emotional expression. Compared to manually designed scoring features, this improves the robustness and generalization ability of singing scores. Therefore, it enhances the applicability, robustness, and generalization ability of singing scores. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating an embodiment of the singing scoring method of this application;

[0011] Figure 2 This is a schematic diagram of a process of one embodiment of the singing scoring method of this application;

[0012] Figure 3 This is a schematic diagram of the framework of an embodiment of the singing scoring device of this application;

[0013] Figure 4 This is a schematic diagram of the framework of an embodiment of the electronic device of this application;

[0014] Figure 5 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0015] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0016] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0017] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the slash " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper indicates two or more objects.

[0018] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the singing scoring method of this application. It should be noted that the process operations in this embodiment can be executed by an electronic device with computing capabilities or related equipment containing an electronic device. The specific structure and type of the electronic device and related equipment containing an electronic device are not limited here. The process operations in this embodiment can be applied to application scenarios such as music teaching, singing entertainment, and singing competitions. The application scenarios of this embodiment are not limited here. Specifically, this embodiment may include the following steps:

[0019] Step S11: Obtain the singing audio to be evaluated and the target sheet music of the target song to which the singing audio belongs.

[0020] In one implementation scenario, as one possible example, the singing audio to be evaluated can be obtained by capturing the audio while the singer is singing the target song a cappella; or, as another possible example, the singing audio to be evaluated can also be obtained by capturing the audio while the singer is singing the target song with accompaniment; or, as yet another possible example, for the former example, after capturing the audio, the captured audio data can be further filtered (such as filtering the accompaniment) to obtain the vocal audio of the singer singing the target song, which can then be used as the singing audio to be evaluated. The above examples are merely a few possible examples of obtaining the singing audio to be evaluated in practical applications; other possible scenarios are not limited here, nor will they be listed one by one.

[0021] In one implementation scenario, the target musical score of the target song may include, but is not limited to, numbered musical notation, staff notation, etc. The type of target musical score is not limited here.

[0022] Step S12: Extract the acoustic features of the singing audio and extract the singing sequence of the target musical score.

[0023] In this embodiment of the disclosure, the singing sequence may include the pitch and beat length of the target song at each singing moment. For example, taking the target musical score as a simplified musical notation, the mapping relationship between simplified musical notation, note name, and MIDI semitone value can be obtained. Based on this, the target musical score can be converted into a singing sequence such as {[70,0.5][72,05],[70,0.5],[74,1]……}, where each square bracket represents a singing moment, the first number in the square bracket represents the pitch, and the second number represents the beat length. In addition, the singing sequence may also include key signature (e.g., 1=C), time signature (e.g., 4 / 4), etc., which are not limited here.

[0024] Furthermore, the acoustic features of singing audio can include, but are not limited to, FBank (Filter Bank) and MFCC (Mel-Frequency Cepstral Coefficients). The specific type of acoustic feature is not limited here. Taking MFCC as an example, the acoustic feature of singing audio can specifically be 40-dimensional Mel-Frequency Cepstral Coefficients, used to capture the spectral information of the singing audio. Of course, acoustic features are not limited to this; other feature dimensions or types of features can also be used. The types and dimensions of acoustic features are not limited here, nor will they be listed individually.

[0025] Step S13: Based on the first extraction model, feature extraction is performed on the acoustic features to obtain the first feature representation, and based on the second extraction model, feature extraction is performed on the singing sequence to obtain the second feature representation.

[0026] In this embodiment of the disclosure, both the first extraction model and the second extraction model can adopt a target network framework, which may include a convolutional neural network and a recurrent neural network connected in sequence. It should be noted that convolutional neural networks can be used to extract local features of audio data, such as notes, melodies, and pitch changes, while recurrent neural networks can be used to capture the temporal dynamic features of audio data, such as rhythm changes and emotional expression.

[0027] In one implementation scenario, please refer to the following: Figure 2 , Figure 2 This is a schematic diagram of the framework of one embodiment of the singing scoring method of this application. Figure 2 As shown, the convolutional neural network in the first extraction model can include multiple two-dimensional convolutional layers (Conv2D) to extract multi-scale features and enhance the model's ability to capture information at different frequencies and time scales, such as... Figure 2 The first convolutional layer has 32 channels (C) and a size (K) of 3. The first convolutional layer has a 7-level kernel, and the second convolutional layer has 64 channels and a size of 3. The convolutional kernel has a size of 7, and the third convolutional layer has 128 channels and a size of 3. A convolutional kernel of size 7. Each convolutional layer can be followed by a pooling layer (e.g., max pooling). Figure 2 As shown, after the first convolutional layer, there can be 2... A max-pooling layer of 4, followed by a 3-layer max-pooling layer. A max pooling layer of 5, followed by a third convolutional layer, can be 3. A max-pooling layer of 5. Furthermore, the recurrent neural network in the first extraction model may include gated recurrent units (GRUs), long short-term memory networks, etc., and there are no restrictions on the recurrent neural network here. For example... Figure 2 As shown, the feature dimension of the hidden layer in a recurrent neural network can be 256; however, no limitation is placed on the feature dimension of the hidden layer features here. Of course, Figure 2 The example shown is merely one possible instance of the first extraction model in practical applications. The specific structure of the first extraction model is not limited here, nor will it be listed in detail.

[0028] In one implementation scenario, please continue to refer to [the relevant documentation]. Figure 2 The convolutional neural network in the second extraction model can include several one-dimensional convolutional layers (Conv 1D), such as... Figure 2 The convolutional neural network in the middle can include a one-dimensional convolutional layer containing a convolutional kernel with 32 channels (C) and a size (K) of 9. Furthermore, the recurrent neural network in the second extraction model can include gated recurrent units, long short-term memory networks, etc., and there is no limitation on the recurrent neural network here. For example... Figure 2 As shown, the hidden layer feature dimension of a recurrent neural network can be 64; however, no limitation is placed on the feature dimension of the hidden layer features here. Of course, Figure 2 The example shown is merely one possible instance of the second extraction model. The specific structure of the second extraction model is not limited here, nor will it be listed in detail.

[0029] In one implementation scenario, as a possible example, a pyramid pooling network can be placed between the convolutional neural network and the recurrent neural network in the target network framework; or, as another possible example, a pyramid pooling network can be placed after the recurrent neural network in the target network framework. It should be noted that the pyramid pooling network is used to extract feature maps at different scales to further enrich the feature representation. Furthermore, for the specific implementation of the pyramid pooling network, please refer to the technical details of pyramid pooling networks, which will not be elaborated here. Of course, the above examples are only a few possible configurations of the pyramid pooling network; the configuration of the pyramid pooling network is not limited here, nor will it be listed in detail.

[0030] In one implementation scenario, the recurrent neural network of the target network framework can also incorporate an attention mechanism. As one possible example, the attention mechanism can include, but is not limited to, self-attention mechanisms, enabling the model to focus on important parts of the input data and improve its ability to capture key features. As another possible example, the attention mechanism can also include, but is not limited to, multi-head attention mechanisms, allowing the model to capture complex relationships in the input data from multiple perspectives, thereby enhancing the model's expressive power.

[0031] In one implementation scenario, the recurrent neural network of the target network framework can include, but is not limited to, bidirectional long short-term memory networks and bidirectional gated recurrent units. It should be noted that bidirectional long short-term memory networks can capture both forward and backward information from the input, improving the ability to model dynamic features over time, while bidirectional gated recurrent units can improve computational complexity, thereby enhancing the model's training efficiency.

[0032] In one implementation scenario, at least one of the convolutional neural networks or recurrent neural networks in the target network framework can use residual connections, which can alleviate the gradient vanishing and gradient exploding problems and improve the stability of model training.

[0033] Step S14: Based on the first feature representation and the second feature representation, make a prediction to obtain the singing score of the singing audio.

[0034] In one implementation scenario, as a possible approach, after obtaining the first feature representation and the second feature representation, they can be fused to obtain a fused feature representation. Then, predictions are made based on this fused feature representation across several evaluation dimensions to obtain sub-scores for each dimension. These sub-scores are then fused to obtain the final performance score. It should be noted that the evaluation dimensions may include, but are not limited to, pitch accuracy, rhythm, and lyrics. Specifically, the pitch accuracy sub-score evaluates the singer's performance in pitch, the rhythm sub-score evaluates their performance in rhythm, and the lyrics sub-score evaluates their pronunciation and clarity. This method, by combining the first and second feature representations to predict scores across several evaluation dimensions, obtains sub-scores for each dimension. This means that each evaluation dimension can share the common feature of the fused feature representation, and the sub-scores from each dimension can be combined to obtain the final performance score, thus improving the comprehensiveness of the performance evaluation.

[0035] In a specific implementation scenario, the first feature representation and the second feature representation can be fused through operations such as concatenation to obtain a fused feature representation.

[0036] In a specific implementation scenario, each evaluation dimension can be designed with an independent output layer to predict scores for different evaluation dimensions and obtain sub-scores for the corresponding evaluation dimensions.

[0037] In a specific implementation scenario, the performance score can be obtained by weighting the sub-scores of each evaluation dimension. As one possible example, weighting coefficients can be pre-set for each evaluation dimension. Based on these pre-set weighting coefficients, the sub-scores of each evaluation dimension can be weighted separately to obtain the performance score. As another possible example, weighting coefficients can be adaptively predicted for each evaluation dimension. Based on these adaptively predicted weighting coefficients, the sub-scores of each evaluation dimension can be weighted separately to obtain the performance score. For example, when making predictions based on fused feature representations, predictions can be made from several evaluation dimensions to obtain the sub-scores and weighting coefficients for each evaluation dimension. Of course, the above examples are only a few possible ways to obtain the weighting coefficients of each evaluation dimension in practical applications. Other possible methods are not limited here, nor will they be listed in detail.

[0038] In another implementation scenario, as an alternative possible implementation method, distinct from the aforementioned implementation methods, please continue to refer to [the relevant documentation / reference]. Figure 2 After obtaining the first and second feature representations, and before predicting sub-scores based on them, the singing audio can be analyzed to obtain a pitch distribution sequence. It should be noted that the pitch distribution sequence can include the duration of each pitch in the singing audio. Similarly, the pitch distribution sequence can also include the percentage of duration at each pitch (i.e., the ratio of the duration of each pitch to the total duration). Furthermore, for ease of visualization, the pitch distribution sequence can also be represented as a pitch histogram, which is not limited here. Based on this, predictions can be made using the first feature representation, the second feature representation, and the pitch distribution sequence to obtain the singing score of the audio. The specific process can be found in the description of score prediction and fusion from several evaluation dimensions in the aforementioned embodiments, and will not be repeated here. The above method analyzes the singing audio to obtain a pitch distribution sequence, which includes the duration of each pitch in the singing audio. Then, based on the first feature representation, the second feature representation, and the pitch distribution sequence, a prediction is made to obtain the singing score of the singing audio. The pitch distribution sequence can be further added as reference information for score prediction, which helps to provide richer reference information for score prediction.

[0039] In one implementation scenario, the singing score can be predicted by a singing scoring model, which may include a first extraction model and a second extraction model. Of course, in addition to these, the singing scoring model may also include other network models, such as an output layer for predicting scores from different evaluation dimensions, which is not limited here. To improve the performance of the singing scoring model, it can be trained before predicting the singing score. Specifically, sample singing audio and the sample target music score of the target song to which the sample singing audio belongs can be obtained. The sample singing audio is labeled with the sample singing score and the type of the target song (e.g., pop, children's song, rock, etc.). Then, the sample acoustic features of the sample singing audio are extracted, and the sample singing sequence of the target music score is extracted. The sample singing sequence may include the sample pitch and sample beat length of the target song at each sample singing moment. It should be noted that the specific meanings of the sample singing audio, the sample acoustic features, the sample target score, and the sample singing sequence can be found in the aforementioned descriptions of the singing audio, acoustic features, target score, and singing sequence, respectively. These will not be elaborated upon here. After obtaining the sample acoustic features of the sample singing audio and the sample singing sequence of the sample target score, feature extraction can be performed based on the first extraction model, yielding a first sample feature representation. Then, feature extraction can be performed on the sample singing sequence based on the second extraction model, yielding a second sample feature representation. Based on these two representations, a prediction can be made to obtain the predicted singing score of the sample singing audio. Furthermore, based on the first feature representation, a prediction can be made to determine the predicted song type of the target song to which the sample singing audio belongs. Finally, the network parameters of the singing scoring model can be adjusted based on the differences between the predicted singing score and the sample singing score, as well as the differences between the predicted song type and the sample song type. The above approach, which designs a branch task for song type prediction during model training to assist the main task of score prediction, helps to improve the performance of the singing scoring model.

[0040] In a specific implementation scenario, before formal training begins, data augmentation can be performed on the sample singing audio to increase the diversity and robustness of the training data. As one possible example, data augmentation techniques such as temporal domain augmentation and frequency domain augmentation can be used to augment the sample singing audio. Another possible example is using generative adversarial networks or other synthetic techniques to generate additional training data to improve the model's generalization ability. As yet another possible example, the sample singing audio can be randomly sliced ​​to increase the diversity of the training samples and alleviate overfitting.

[0041] In a specific implementation scenario, before formally starting training, preprocessing operations such as noise reduction and energy normalization can be performed on the sample singing audio. For example, noise reduction algorithms can be applied to the sample singing audio to eliminate background noise as much as possible and improve the clarity of the audio signal. As another example, energy normalization can be performed on the audio signal to ensure that its amplitude range remains consistent as much as possible, reducing the impact of different recording devices. It should be noted that the above preprocessing operations can also be performed during the model inference phase.

[0042] In a specific implementation scenario, at least one of the convolutional neural networks and recurrent neural networks can also introduce a Dropout layer (e.g., the ratio can be set to 0.1) to prevent overfitting of the training data as much as possible during the training process, which helps to improve the generalization performance of the model.

[0043] In a specific implementation scenario, similar to how the aforementioned singing score includes sub-scores of several evaluation dimensions, the predicted singing score of the sample singing audio can also include first sub-scores of several evaluation dimensions, and the sample singing score can also include second sub-scores of several evaluation dimensions. The difference between the predicted singing score and the sample singing score can then be obtained by weighting the differences between the first and second sub-scores of each evaluation dimension. For example, this difference can be measured using the mean squared error loss function. For ease of description, it can be denoted as L. mse For specific calculation methods, please refer to the technical details of the mean squared error loss function, which will not be elaborated here.

[0044] In one specific implementation, when making predictions based on the first sample feature representation, the predicted probability values ​​of the target songs belonging to the sample singing audio for each preset song type can be obtained (the predicted song type can be the preset song type corresponding to the highest predicted probability value). Then, a loss metric can be applied to the sample song type and the predicted probability values ​​of each preset song type using the cross-entropy loss function to measure the difference between the predicted song type and the sample song type. For ease of description, taking the cross-entropy loss function as an example, the above difference can be expressed as L. ce :

[0045] ,

[0046] In the above formula, K represents the total number of preset song types, p k This represents the predicted probability value for the k-th preset song type.

[0047] In a specific implementation scenario, the Adam optimizer can be used for model training to improve convergence speed and stability. The specific process of adjusting parameters based on the aforementioned differences can be found in the technical details of the Adam optimizer, and will not be elaborated upon here. Furthermore, the learning rate for model training can be adaptively adjusted, such as using cosine annealing or gradual decay strategies to optimize the model training process.

[0048] In a specific implementation scenario, before adjusting the parameters, any sample singing audio can be selected as the first singing audio, and a sample singing audio different from the first singing audio can be selected as the second singing audio. Then, the first score difference between the sample singing scores of the first and second singing audios is obtained, and the second score difference between the predicted singing scores of the first and second singing audios is obtained. Based on the differences between the predicted singing scores and the sample singing scores, the differences between the predicted song type and the sample song type, and the differences between the first and second score differences, the network parameters of the singing scoring model can be adjusted. For ease of description, the total loss L, which includes the above three types of differences, is... total It can be represented as:

[0049] ,

[0050] In the above formula, L mse L represents the difference between the predicted singing score and the sample singing score. ce L represents the difference between the predicted song type and the sample song type. diff-consistence This represents the difference between the first score difference and the second score difference. For ease of understanding, let's take the example of selecting the i-th sample singing audio as the first singing audio and the j-th sample singing audio as the second singing audio. The difference L between the first and second score differences is... diff-consistence It can be represented as:

[0051] ,

[0052] In the above formula, N represents the total number of sample singing audios, abs represents the absolute value, and y i y j Let represent the sample singing scores of the first singing audio (i.e., the i-th sample singing audio) and the second singing audio (i.e., the j-th sample singing audio), respectively. Let $\frac{i}{j}$ represent the sample singing score of the first singing audio (i.e., the i-th sample singing audio) and the predicted singing score of the second singing audio (i.e., the j-th sample singing audio), respectively. The above method, by further incorporating the difference between the first score difference between the sample singing scores of the first and second singing audios, and the difference between the second score difference between the predicted singing scores of the first and second singing audios, introduces a score difference consistency loss. This helps the model pay more attention to the differences in singing levels and improves the discriminative power of the predicted scores.

[0053] In one implementation scenario, before adjusting the parameters, a transformation can be performed based on the first sample feature representation to obtain the query sample feature representation (e.g., it can be denoted as Q). audio Based on the second sample feature representation, a transformation is performed to obtain the key sample feature representation (e.g., it can be denoted as K). score ) and value sample feature representation (e.g., can be denoted as V) score It should be noted that the query sample feature representation includes the query sample sub-features of each section, the key sample feature representation includes the key sample sub-features of each section, and the value sample feature representation includes the value sample sub-features of each section. Based on this, the query sample feature representation and the key sample feature representation can interact to obtain the sample weight values ​​of each section. Then, the value sample sub-features and weights of each section can be weighted to obtain the weighted sample sub-features of each section. It should be noted that the sample weight value of a section can represent the degree of consistency between the sample singing audio and the sample target musical score at the section level. The fused sample feature representation includes the weighted sample sub-features of each section. For example, a dot product operation can be performed based on the query sample feature representation and the key sample feature representation to obtain the sample weight values ​​of each section. For ease of description, the sample weight value of the j-th section can be denoted as β. j Then the weighted sample sub-feature D of the j-th subsection j It can be represented as:

[0054] D j =β j V score,j

[0055] In the above formula, V score,j Let L represent the value sample sub-feature of the j-th section. Based on this, the network parameters of the singing scoring model can be adjusted according to the differences between the predicted singing score and the sample singing score, the differences between the predicted song type and the sample song type, and the differences between the weighted sample sub-features of each section and the query sample sub-features (e.g., measured using the mean squared error loss function). In other words, in this case, the total loss L, which includes the above three types of differences, is... total It can be represented as:

[0056] ,

[0057] In the above formula, L mse1 L represents the difference between the predicted singing score and the sample singing score. ce L represents the difference between the predicted song type and the sample song type. mse2 This represents the difference between the weighted sample sub-features of each measure and the query sample sub-features. By further incorporating the difference between the weighted sample sub-features of each measure and the query sample sub-features when measuring the training loss, the model can learn the ideal performance features under the constraints of the target score during training. This forces the model to learn to score the actual performance based on the ideal performance features under the constraints of the target score.

[0058] It should be noted that the above example only represents the total loss L in practical applications. total Several possible examples, here for the total loss L total Other possible scenarios are not limited, such as total loss L. total It can also include the difference L between the predicted singing score and the sample singing score. mse1 The difference between the predicted song type and the sample song type L ce The difference L between the first score difference and the second score difference diff-consistence The difference between the weighted sample sub-features of each section and the query sample sub-features (L) mse2 Examples will not be listed here.

[0059] In one implementation scenario, during the training phase, multiple singing scoring models can be trained based on sample singing audio and their target musical scores. The final performance score can be obtained by fusing (e.g., averaging) the scores predicted by each of the individual models, thus improving the stability of the scoring. It should be noted that each singing scoring model can use the same network structure (e.g., all models can use...). Figure 2 The network structure shown may differ between singing scoring models. Furthermore, the training data used by each singing scoring model may be different, or the training data used may be the same. Additionally, the hyperparameters (such as learning rate, number of training iterations, etc.) used by each singing scoring model during training may be the same, or the hyperparameters used may be different. Of course, the above examples are merely a few possible examples of the strategies employed by different singing scoring models during training; other possible scenarios are not limited here, nor will they be listed in detail.

[0060] The above scheme obtains the singing audio to be evaluated and the target score of the target song to which the singing audio belongs. Then, it extracts the acoustic features of the singing audio and the singing sequence of the target score, including the pitch and beat length of the target song at each moment of performance. Based on a first extraction model, it extracts features from the acoustic features to obtain a first feature representation. Then, based on a second extraction model, it extracts features from the singing sequence to obtain a second feature representation. Both the first and second extraction models use a target network framework, which includes sequentially connected convolutional neural networks and recurrent neural networks. Finally, it predicts the singing score based on the first and second feature representations. This approach directly extracts the corresponding feature representations from the singing audio and the target score. This method of score prediction, compared to rule-based scoring, is more adaptable to the performance differences of various singers, thus improving the applicability of singing scores. Furthermore, because the score prediction process references not only the primary feature representation extracted from the singing audio but also the target score of the corresponding song, using the singing sequence to aid prediction, it helps improve the accuracy of singing scores. Additionally, the feature extraction process combines convolutional neural networks and recurrent neural networks. The former focuses on local features such as notes, melody, and pitch changes, while the latter captures temporal dynamic features such as rhythm changes and emotional expression. Compared to manually designed scoring features, this improves the robustness and generalization ability of singing scores. Therefore, it enhances the applicability, robustness, and generalization ability of singing scores.

[0061] Please see Figure 3 , Figure 3 This is a schematic diagram of the framework of an embodiment of the singing scoring device of this application. The singing scoring device 30 includes: an acquisition and preparation module 31, a content extraction module 32, a feature extraction module 33, and a score prediction module 34. The acquisition and preparation module 31 is used to acquire the singing audio to be evaluated and the target score of the target song to which the singing audio belongs. The content extraction module 32 is used to extract the acoustic features of the singing audio and extract the singing sequence of the target score. The singing sequence includes the pitch and beat length of the target song at each singing moment. The feature extraction module 33 is used to extract features from the acoustic features based on a first extraction model to obtain a first feature representation, and to extract features from the singing sequence based on a second extraction model to obtain a second feature representation. Both the first extraction model and the second extraction model adopt a target network framework, which includes a sequentially connected convolutional neural network and a recurrent neural network. The score prediction module 34 is used to predict the singing score of the singing audio based on the first feature representation and the second feature representation.

[0062] In the above scheme, the singing scoring device 30 acquires the singing audio to be evaluated and the target score of the target song to which the singing audio belongs. It then extracts the acoustic features of the singing audio and the singing sequence of the target score, including the pitch and beat length of the target song at each singing moment. Based on a first extraction model, it extracts features from the acoustic features to obtain a first feature representation. Based on a second extraction model, it extracts features from the singing sequence to obtain a second feature representation. Both the first and second extraction models use a target network framework, which includes sequentially connected convolutional neural networks and recurrent neural networks. Finally, based on the first and second feature representations, it predicts and obtains the singing score of the singing audio. This approach directly extracts the corresponding features from the singing audio and the target score. Feature representation is used to predict scores, which, compared to rule-based scoring, can better adapt to the performance differences of different singers, thus improving the applicability of singing scores. Furthermore, since the score prediction process references not only the first feature representation extracted from the singing audio but also the target score of the corresponding song, using the singing sequence to assist in prediction, it helps improve the accuracy of singing scores. Moreover, the feature extraction process combines the target network architecture of convolutional neural networks and recurrent neural networks. The former focuses on local features such as notes, melody, and pitch changes, while the latter captures temporal dynamic features such as rhythm changes and emotional expression. Compared to manually designed scoring features, this improves the robustness and generalization ability of singing scores. Therefore, it enhances the applicability, robustness, and generalization ability of singing scores.

[0063] In some disclosed embodiments, the score prediction module 34 includes a feature fusion submodule for fusing based on a first feature representation and a second feature representation to obtain a fused feature representation; the score prediction module 34 includes a dimension evaluation submodule for predicting from several evaluation dimensions based on the fused feature representation to obtain sub-scores for each evaluation dimension; and the score prediction module 34 includes a score fusion submodule for fusing based on the sub-scores for each evaluation dimension to obtain a performance score.

[0064] In some disclosed embodiments, several evaluation dimensions include at least one of pitch dimension, rhythm dimension, and lyrics dimension; and / or, the performance score is obtained by weighting the sub-scores of each evaluation dimension.

[0065] In some disclosed embodiments, the singing scoring device 30 includes a sequence acquisition module for analyzing the singing audio to obtain a pitch distribution sequence; wherein the pitch distribution sequence includes the duration of the singing audio at each pitch; and the score prediction module 34 is specifically used to predict the singing score of the singing audio based on the first feature representation, the second feature representation and the pitch distribution sequence.

[0066] In some disclosed embodiments, the singing score is predicted by a singing scoring model, which includes a first extraction model and a second extraction model. The singing scoring device 30 includes a sample acquisition module for acquiring sample singing audio and the sample target music score of the sample target song to which the sample singing audio belongs; wherein, the sample singing audio is labeled with a sample singing score and the sample song type of the sample target song to which it belongs; the singing scoring device 30 includes a sample extraction module for extracting sample acoustic features of the sample singing audio and extracting the sample singing sequence of the sample target music score; wherein, the sample singing sequence includes the sample pitch and sample beat length of the sample target song at each sample singing moment; the singing scoring device 30 includes a sample feature module. The system is used to extract features from the acoustic features of the sample based on the first extraction model to obtain a first sample feature representation, and to extract features from the sample singing sequence based on the second extraction model to obtain a second sample feature representation; the singing scoring device 30 includes a sample prediction module, used to predict the predicted singing score of the sample singing audio based on the first sample feature representation and the second sample feature representation, and to predict the predicted song type of the sample target song to which the sample singing audio belongs based on the first sample feature representation; the singing scoring device 30 includes a parameter adjustment module, used to adjust the network parameters of the singing scoring model based on the difference between the predicted singing score and the sample singing score, and the difference between the predicted song type and the sample song type.

[0067] In some disclosed embodiments, the singing scoring device 30 includes an audio selection module for selecting any sample singing audio as the first singing audio and selecting a sample singing audio different from the first singing audio as the second singing audio; the singing scoring device 30 includes a difference calculation module for obtaining a first score difference between the sample singing scores of the first singing audio and the second singing audio, and obtaining a second score difference between the predicted singing scores of the first singing audio and the second singing audio; the parameter adjustment module is specifically used to adjust the network parameters of the singing scoring model based on the difference between the predicted singing score and the sample singing score, the difference between the predicted song type and the sample song type, and the difference between the first score difference and the second score difference.

[0068] In some disclosed embodiments, during the training phase, multiple singing scoring models are trained based on sample singing audio and sample target musical scores, and the singing score of the singing audio is obtained by fusing the singing scores predicted by each singing scoring model; and / or, the predicted singing score of the sample singing audio includes a first sub-score of several evaluation dimensions, the sample singing score includes a second sub-score of several evaluation dimensions, and the difference between the predicted singing score and the sample singing score is obtained by weighting the differences between the first sub-score and the second sub-score of each evaluation dimension.

[0069] In some disclosed embodiments, the predicted singing score is obtained by predicting a fused sample feature representation of a first sample feature representation and a second sample feature representation. The singing scoring device 30 includes a feature transformation module, used to transform based on the first sample feature representation to obtain a query sample feature representation, and to transform based on the second sample feature representation to obtain a key sample feature representation and a value sample feature representation; wherein, the query sample feature representation includes query sample sub-features of each measure, the key sample feature representation includes key sample sub-features of each measure, and the value sample feature representation includes value sample sub-features of each measure; the singing scoring device 30 includes a feature interaction module, used to interact based on the query sample feature representation and the key sample feature representation. The interaction between the features represents the sample weight values ​​of each section; the sample weight value of each section represents the degree of consistency between the sample singing audio and the sample target musical score at the section level; the singing scoring device 30 includes a feature weighting module, which is used to weight the value sample sub-features and weights of each section to obtain the weighted sample sub-features of each section; wherein, the fused sample feature represents the weighted sample sub-features of each section; the parameter adjustment module is specifically used to adjust the network parameters of the singing scoring model based on the differences between the predicted singing score and the sample singing score, the differences between the predicted song type and the sample song type, and the differences between the weighted sample sub-features of each section and the query sample sub-features.

[0070] In some disclosed embodiments, at least one of the convolutional neural network and the recurrent neural network introduces a Dropout layer; and / or, the target network framework includes a pyramid pooling network following the recurrent neural network; and / or, the recurrent neural network introduces an autonomous mechanism; and / or, the recurrent neural network includes either a bidirectional long short-term memory network or a bidirectional gated recurrent unit; and / or, at least one of the convolutional neural network and the recurrent neural network employs residual connections.

[0071] Please see Figure 4 , Figure 4 This is a schematic diagram of a framework of an embodiment of the electronic device of this application. The electronic device 40 includes at least a memory 41 and a processor 42 coupled to each other. The memory 41 stores at least program instructions, and the processor 42 is used to execute the program instructions to implement the steps in any of the above-described singing scoring method embodiments. For details, please refer to the foregoing disclosed embodiments, which will not be repeated here.

[0072] Specifically, processor 42 controls itself and memory 41 to implement the steps in any of the singing scoring method embodiments described above. Processor 42 can also be referred to as a CPU (Central Processing Unit). Processor 42 may be an integrated circuit chip with signal processing capabilities. Processor 42 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 42 can be implemented using integrated circuit chips.

[0073] The above scheme involves an electronic device 40 acquiring the singing audio to be evaluated and the target score of the target song to which the singing audio belongs. It then extracts the acoustic features of the singing audio and the singing sequence of the target score, including the pitch and beat length of the target song at each singing moment. Based on a first extraction model, it extracts features from the acoustic features to obtain a first feature representation. Based on a second extraction model, it extracts features from the singing sequence to obtain a second feature representation. Both the first and second extraction models use a target network framework, which includes sequentially connected convolutional neural networks and recurrent neural networks. Finally, based on the first and second feature representations, it predicts and obtains the singing score of the singing audio. This approach directly extracts the corresponding features from the singing audio and the target score. The feature representation used for score prediction, compared to rule-based scoring, is more adaptable to the performance differences of various singers, thus improving the applicability of singing scores. Furthermore, since the score prediction process references not only the first feature representation extracted from the singing audio but also the target score of the corresponding song, using the singing sequence to aid prediction, it helps improve the accuracy of singing scores. Additionally, the feature extraction process combines convolutional neural networks and recurrent neural networks. The former focuses on local features such as notes, melody, and pitch changes, while the latter captures temporal dynamic features such as rhythm changes and emotional expression. Compared to manually designed scoring features, this improves the robustness and generalization ability of singing scores. Therefore, it enhances the applicability, robustness, and generalization ability of singing scores.

[0074] Please see Figure 5 , Figure 5This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 50 stores program instructions 51 that can be executed by a processor. The program instructions 51 are used to implement the steps in any of the above-described embodiments of the singing scoring method.

[0075] The above scheme involves a computer-readable storage medium 50 acquiring the singing audio to be evaluated and the target score of the target song to which the singing audio belongs. Then, it extracts the acoustic features of the singing audio and the singing sequence of the target score. The singing sequence includes the pitch and beat length of the target song at each singing moment. Based on a first extraction model, it extracts features from the acoustic features to obtain a first feature representation. Based on a second extraction model, it extracts features from the singing sequence to obtain a second feature representation. Both the first and second extraction models use a target network framework, which includes sequentially connected convolutional neural networks and recurrent neural networks. Finally, it predicts the singing score based on the first and second feature representations. This approach, on the one hand, directly extracts features from the singing audio and the target score. The use of feature representations for score prediction, compared to rule-based scoring, is more adaptable to the performance differences of various singers, thus improving the applicability of singing scores. Furthermore, since the score prediction process references not only the first feature representation extracted from the singing audio but also the target score of the corresponding song, using the singing sequence to aid prediction, it helps improve the accuracy of singing scores. Additionally, the feature extraction process combines convolutional neural networks and recurrent neural networks. The former focuses on local features such as notes, melody, and pitch changes, while the latter captures temporal dynamic features such as rhythm changes and emotional expression. Compared to manually designed scoring features, this improves the robustness and generalization ability of singing scores. Therefore, it enhances the applicability, robustness, and generalization ability of singing scores.

[0076] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0077] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0078] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0079] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0080] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0081] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0082] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. A singing scoring method, characterized in that, include: Obtain the singing audio to be evaluated and the target sheet music of the target song to which the singing audio belongs; The acoustic features of the singing audio are extracted, and the singing sequence of the target musical score is extracted; wherein, the singing sequence includes the pitch and beat length of the target song at each singing moment in sequence; Based on the first extraction model, feature extraction is performed on the acoustic features to obtain a first feature representation, and based on the second extraction model, feature extraction is performed on the singing sequence to obtain a second feature representation; wherein, both the first extraction model and the second extraction model adopt a target network framework, and the target network framework includes a sequentially connected convolutional neural network and a recurrent neural network; A prediction is made based on the first feature representation and the second feature representation to obtain a singing score for the singing audio; wherein, the prediction based on the first feature representation and the second feature representation to obtain a singing score for the singing audio includes: The first feature representation and the second feature representation are fused to obtain a fused feature representation; Based on the fusion feature representation, predictions are made from several evaluation dimensions to obtain sub-scores for each evaluation dimension; The performance score is obtained by fusing the sub-scores of each of the evaluation dimensions.

2. The method according to claim 1, characterized in that, The evaluation dimensions include at least one of pitch dimension, rhythm dimension, and lyric dimension; And / or, the performance score is obtained by weighting the sub-scores of each of the evaluation dimensions.

3. The method according to claim 1, characterized in that, Before predicting the singing score of the singing audio based on the first feature representation and the second feature representation, the method further includes: Based on the analysis of the singing audio, a pitch distribution sequence is obtained; wherein, the pitch distribution sequence includes the duration of the singing audio at each pitch; The step of predicting the singing score of the singing audio based on the first feature representation and the second feature representation includes: Based on the first feature representation, the second feature representation, and the pitch distribution sequence, a singing score is obtained from the singing audio.

4. The method according to claim 1, characterized in that, The singing score is predicted by a singing scoring model, which includes a first extraction model and a second extraction model. The training steps of the singing scoring model include: Obtain the sample singing audio and the sample target music score of the sample target song to which the sample singing audio belongs; wherein, the sample singing audio is labeled with the sample singing score and the sample song type to which the sample target song belongs; Extract the acoustic features of the sample singing audio and extract the sample singing sequence of the sample target music score; wherein, the sample singing sequence includes the sample pitch and sample beat length of the sample target song at each sample singing moment; Based on the first extraction model, feature extraction is performed on the acoustic features of the sample to obtain a first sample feature representation, and based on the second extraction model, feature extraction is performed on the singing sequence of the sample to obtain a second sample feature representation; Based on the first sample feature representation and the second sample feature representation, a prediction score for the sample singing audio is obtained, and based on the first sample feature representation, a prediction song type for the sample target song to which the sample singing audio belongs is obtained. Based on the difference between the predicted singing score and the sample singing score, and the difference between the predicted song type and the sample song type, the network parameters of the singing scoring model are adjusted.

5. The method according to claim 4, characterized in that, Before adjusting the network parameters of the singing scoring model based on the difference between the predicted singing score and the sample singing score, and the difference between the predicted song type and the sample song type, the method further includes: Select any of the sample singing audios as the first singing audio, and select a sample singing audio that is different from the first singing audio as the second singing audio; Obtain a first score difference between the sample singing scores of the first singing audio and the second singing audio, and obtain a second score difference between the predicted singing scores of the first singing audio and the second singing audio. The step of adjusting the network parameters of the singing scoring model based on the difference between the predicted singing score and the sample singing score, and the difference between the predicted song type and the sample song type, includes: The network parameters of the singing scoring model are adjusted based on the differences between the predicted singing score and the sample singing score, the differences between the predicted song type and the sample song type, and the differences between the first score difference and the second score difference.

6. The method according to claim 4, characterized in that, During the training phase, multiple singing scoring models are trained based on the sample singing audio and the sample target musical score, and the singing score of the singing audio is obtained by fusing the singing scores predicted by each of the singing scoring models. And / or, the predicted singing score of the sample singing audio includes a first sub-score of several evaluation dimensions, the sample singing score includes a second sub-score of the several evaluation dimensions, and the difference between the predicted singing score and the sample singing score is obtained by weighting the differences between the first sub-scores and the second sub-scores of each of the evaluation dimensions.

7. The method according to claim 4, characterized in that, The predicted singing score is obtained by predicting a fused sample feature representation of the first sample feature representation and the second sample feature representation. Before adjusting the network parameters of the singing scoring model based on the difference between the predicted singing score and the sample singing score, and the difference between the predicted song type and the sample song type, the method further includes: Based on the first sample feature representation, a query sample feature representation is obtained by transformation, and based on the second sample feature representation, a key sample feature representation and a value sample feature representation are obtained by transformation; wherein, the query sample feature representation includes query sample sub-features of each section, the key sample feature representation includes key sample sub-features of each section, and the value sample feature representation includes value sample sub-features of each section; Based on the interaction between the query sample feature representation and the key sample feature representation, the sample weight value of each section is obtained; wherein, the sample weight value of the section represents the degree of consistency between the sample singing audio and the sample target musical score in that section; The weighted sample sub-features of each subsection are obtained by weighting the value sample sub-features and weights of each subsection; wherein, the fused sample feature represents the weighted sample sub-features of each subsection. The step of adjusting the network parameters of the singing scoring model based on the difference between the predicted singing score and the sample singing score, and the difference between the predicted song type and the sample song type, includes: Based on the differences between the predicted singing score and the sample singing score, the differences between the predicted song type and the sample song type, and the differences between the weighted sample sub-features of each section and the query sample sub-features, the network parameters of the singing scoring model are adjusted.

8. The method according to any one of claims 1 to 7, characterized in that, At least one of the convolutional neural network and the recurrent neural network introduces a Dropout layer; And / or, the target network framework includes a pyramid pooling network following the recurrent neural network; And / or, the recurrent neural network incorporates an autonomous mechanism; And / or, the recurrent neural network includes either a bidirectional long short-term memory network or a bidirectional gated recurrent unit; And / or, at least one of the convolutional neural network and the recurrent neural network employs residual connections.

9. A singing scoring device, characterized in that, include: The preparation module is used to acquire the singing audio to be evaluated and the target sheet music of the target song to which the singing audio belongs; The content extraction module is used to extract the acoustic features of the singing audio and extract the singing sequence of the target musical score; wherein, the singing sequence includes the pitch and beat length of the target song at each singing moment in sequence; The feature extraction module is used to extract features from the acoustic features based on a first extraction model to obtain a first feature representation, and to extract features from the singing sequence based on a second extraction model to obtain a second feature representation; wherein, both the first extraction model and the second extraction model adopt a target network framework, and the target network framework includes a sequentially connected convolutional neural network and a recurrent neural network; A score prediction module is used to predict the singing score of the singing audio based on the first feature representation and the second feature representation; wherein, the prediction based on the first feature representation and the second feature representation to obtain the singing score of the singing audio includes: The first feature representation and the second feature representation are fused to obtain a fused feature representation; Based on the fusion feature representation, predictions are made from several evaluation dimensions to obtain sub-scores for each evaluation dimension; The performance score is obtained by fusing the sub-scores of each of the evaluation dimensions.

10. An electronic device, characterized in that, It includes at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor executes the program instructions to implement the singing scoring method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The device stores program instructions that can be executed by a processor, the program instructions being used to implement the singing scoring method according to any one of claims 1 to 8.

Images