Singing evaluation method, apparatus, storage medium, and computing device

Abstract

Embodiments of the present disclosure provide a singing evaluation method and device, a storage medium and a computing device. The method comprises: performing feature extraction on obtained singing audio to obtain a sound spectrum feature of the singing audio; inputting the sound spectrum feature into a multi-scale network model for calculation; wherein the multi-scale network model comprises a model based on a convolutional neural network, and a convolutional layer in the multi-scale network model comprises convolutional kernels corresponding to multiple time scales; and determining a final evaluation result based on a probability distribution of different evaluation results output by the multi-scale network model.

Description

Singing evaluation methods, devices, storage media and computing equipment Technical Field

[0001] The embodiments of this disclosure relate to the field of computer technology, and more specifically, the embodiments of this disclosure relate to a singing evaluation method, apparatus, storage medium, and computing device. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this disclosure as set forth in the specification. The description herein is not intended to be a prior art simply because it is included in this section.

[0003] In recent years, with the emergence of singing software, the demand for singing evaluation has been growing.

[0004] In related technologies, singing evaluation schemes generally rely on pre-set "reference information," such as the MIDI score of a song or sample audio recordings sung by professional singers. Specifically, after acquiring the singing audio, the quality of the singing is measured by comparing the degree of difference between the singing audio and the reference information. Simply put, the more similar the singing audio is to the reference information, the better the singing sample is considered, and vice versa.

[0005] However, this singing evaluation scheme can only be implemented when there is reference information, and cannot be applied in many scenarios without reference information, thus limiting the application scenarios of singing evaluation. Summary of the Invention

[0006] In a first aspect of this disclosure, a singing evaluation method is provided, comprising:

[0007] Feature extraction is performed on the acquired singing audio to obtain the spectral features of the singing audio;

[0008] The acoustic spectrum features are input into a multi-scale network model for calculation; wherein, the multi-scale network model includes a model based on a convolutional neural network, and the convolutional layers in the multi-scale network model include convolutional kernels corresponding to multiple time scales;

[0009] The final evaluation result is determined based on the probability distribution of different evaluation results output by the multi-scale network model; wherein the calculation process of the multi-scale network model includes:

[0010] The acoustic spectrum features are downsampled at multiple time scales along the time dimension to obtain acoustic spectrum sub-feature sequences corresponding to each time scale; wherein, the acoustic spectrum sub-feature sequence is composed of several acoustic spectrum sub-features obtained by downsampling at the same time scale in the order of sampling.

[0011] The acoustic spectrum feature sequence is input into the convolution kernel of the corresponding time scale for convolution calculation to obtain the time scale feature output by the convolution kernel;

[0012] The time-scale features of the multiple time scales are fused to obtain fused features;

[0013] The fused features are input into a multi-classifier for calculation to obtain the probability distribution of the different evaluation results.

[0014] Optionally, the step of extracting features from the acquired singing audio to obtain the spectral features of the singing audio includes:

[0015] The acquired singing audio is converted from the time domain to the frequency domain to obtain the spectral characteristics of the singing audio in the time-frequency domain; wherein, the spectral characteristics in the time-frequency domain include time information in the time domain and frequency information in the frequency domain.

[0016] Optionally, the step of performing a time-domain to frequency-domain conversion on the acquired singing audio to obtain the spectral characteristics of the singing audio in the time-frequency domain includes:

[0017] The acquired singing audio is subjected to constant Q transformation to obtain the spectrogram features of the singing audio in the time-frequency domain;

[0018] Alternatively, the acquired singing audio can be processed by a short-time Fourier transform, and the transformed features can be converted into Mel-scale spectral features using a Mel-scale filter.

[0019] Optionally, the downsampling of the spectral features at multiple time scales in the time dimension includes:

[0020] For a spectral feature of size F*T, the spectral feature is downsampled in the time dimension at time intervals of T / 2, ..., T / (2^n); where F is the frequency amplitude of the spectral feature, T is the duration of the spectral feature, and n is a positive integer.

[0021] Optionally, the convolution kernels corresponding to the various time scales are of the same size.

[0022] Optionally, before downsampling the spectral features at the multiple time scales in the time dimension, the method further includes:

[0023] The spectral features are downsampled at various time scales along the frequency dimension.

[0024] Optionally, the step of fusing the time-scale features of the multiple time scales to obtain fused features includes:

[0025] Multiple time-scale features are upsampled in the time dimension to obtain a second time-scale feature with the same original time scale as the spectral feature;

[0026] Multiple second-time features are concatenated and then convolved using a convolution kernel for fusion to obtain fused features.

[0027] Optionally, after obtaining the fusion features, the following may also be included:

[0028] Determine if the current loop count has reached the preset number of iterations;

[0029] If the preset number of iterations is not reached, the current loop count is incremented by 1, and the fused feature is used as the new spectrogram feature. The process of downsampling the spectrogram feature at multiple time scales in the time dimension is repeated until the current loop reaches the preset number of iterations.

[0030] Optional, also includes:

[0031] The scoring score of the singing audio is calculated based on the probability distribution of the different evaluation results.

[0032] Optionally, calculating the score of the singing audio based on the probability distribution of the different evaluation results includes:

[0033] Obtain the preset weight for each evaluation result;

[0034] The probability value of each scoring result in the probability distribution is multiplied by the corresponding preset weight, and the products of each scoring result are added together. The sum is determined as the score of the singing audio.

[0035] Optionally, before inputting the spectral features into the multi-scale network model for computation, the method further includes:

[0036] The acoustic spectrum features are divided into several short acoustic spectrum features according to a preset duration;

[0037] The step of inputting the acoustic spectrum features into a multi-scale network model for calculation includes:

[0038] The short spectral features of the aforementioned segments are sequentially input into a multi-scale network model for calculation;

[0039] After calculating the score corresponding to each short spectral feature, the score corresponding to each short spectral feature is constructed into a score curve of the singing audio changing over time according to the chronological order of the short spectral features.

[0040] Optionally, the classification labels of the training samples of the multi-scale network model include the labels of the different evaluation results;

[0041] When training the model based on the training samples, the multi-scale network model uses a multi-classification loss function to calculate the loss and performs backpropagation based on the loss results to update the model parameters.

[0042] Optionally, the different evaluation results include at least three different evaluation results;

[0043] The determination of the final evaluation result based on the probability distribution of different evaluation results output by the multi-scale network model includes:

[0044] Obtain the probability distributions of at least three different evaluation results output by the multi-scale network model, determine the largest probability value in the probability distribution, and determine the evaluation result corresponding to the largest probability value as the final evaluation result.

[0045] In a second aspect of this disclosure, a singing evaluation device is provided, the device comprising:

[0046] The extraction unit performs feature extraction on the acquired singing audio to obtain the spectral features of the singing audio;

[0047] The input unit inputs the spectral features into a multi-scale network model for calculation; wherein, the multi-scale network model includes a model based on a convolutional neural network, and the convolutional layers in the multi-scale network model include convolutional kernels corresponding to multiple time scales;

[0048] The determination unit determines the final evaluation result based on the probability distribution of different evaluation results output by the multi-scale network model; wherein the calculation process of the multi-scale network model includes the following units:

[0049] The sampling unit downsamples the acoustic spectrum features at multiple time scales along the time dimension to obtain acoustic spectrum sub-feature sequences corresponding to each time scale; wherein, the acoustic spectrum sub-feature sequence is composed of several acoustic spectrum sub-features obtained by downsampling at the same time scale in the order of sampling.

[0050] The computing unit inputs the acoustic spectrum feature sequence into the convolution kernel of the corresponding time scale for convolution calculation to obtain the time scale features output by the convolution kernel;

[0051] The fusion unit fuses the time-scale features of the multiple time scales to obtain fused features;

[0052] The classification unit inputs the fused features into a multi-classifier for calculation to obtain the probability distribution of the different evaluation results.

[0053] Optionally, the extraction unit is further configured to perform time-domain to frequency-domain conversion processing on the acquired singing audio to obtain the spectral characteristics of the singing audio in the time-frequency domain; wherein, the spectral characteristics in the time-frequency domain include time information in the time domain and frequency information in the frequency domain.

[0054] Optionally, the extraction unit further includes:

[0055] The first processing subunit performs constant Q-transformation on the acquired singing audio to obtain the spectrogram features of the singing audio in the time-frequency domain;

[0056] Alternatively, the second processing subunit performs a short-time Fourier transform on the acquired singing audio and converts the transformed features into Mel-scale spectral features using a Mel-scale filter.

[0057] Optionally, the sampling unit is further configured to downsample the spectral feature of size F*T in the time dimension for time intervals of T / 2, ..., T / (2^n); where F is the frequency amplitude of the spectral feature, T is the duration of the spectral feature, and n is a positive integer.

[0058] Optionally, the convolution kernels corresponding to the various time scales are of the same size.

[0059] Optionally, the sampling unit is further configured to downsample the acoustic spectral features at the multiple time scales in the frequency dimension.

[0060] Optionally, the fusion unit further includes:

[0061] The first fusion subunit upsamples multiple time-scale features in the time dimension to obtain a second time-scale feature that is the same as the original time-scale of the acoustic spectrum feature.

[0062] The second fusion subunit connects multiple second temporal features and performs convolution calculations using a convolution kernel for fusion to obtain fused features.

[0063] Optionally, following the fusion unit, the following may also be included:

[0064] The loop unit determines whether the current loop count has reached the preset number of times; if the preset number of times has not been reached, the current loop count is incremented by 1, and the fused feature is used as the new spectral feature. The sampling unit is then re-executed until the current loop count reaches the preset number of times.

[0065] Optional, also includes:

[0066] The scoring unit calculates the score of the singing audio based on the probability distribution of the different evaluation results.

[0067] Optionally, the scoring unit further includes:

[0068] Obtain the sub-unit and the preset weight for each evaluation result;

[0069] The process involves multiplying the probability value of each scoring result in the probability distribution by its corresponding preset weight, summing the products of each scoring result, and determining the summed result as the score of the singing audio.

[0070] Optionally, before the input unit, the following may also be included:

[0071] The segmentation unit divides the spectral features into several short spectral features according to a preset duration;

[0072] The input unit is further used to sequentially input the short spectral features of the several segments into the multi-scale network model for calculation;

[0073] Following the scoring unit, the system further includes:

[0074] The construction unit, after calculating the score corresponding to each short spectral feature, constructs a score curve of the singing audio changing over time according to the temporal order of the short spectral features.

[0075] Optionally, the classification labels of the training samples of the multi-scale network model include the labels of the different evaluation results;

[0076] When training the model based on the training samples, the multi-scale network model uses a multi-classification loss function to calculate the loss and performs backpropagation based on the loss results to update the model parameters.

[0077] Optionally, the different evaluation results include at least three different evaluation results;

[0078] The determining unit is further configured to obtain the probability distribution of at least three different evaluation results output by the multi-scale network model, determine the largest probability value in the probability distribution, and determine the evaluation result corresponding to the largest probability value as the final evaluation result.

[0079] In a third aspect of this disclosure, a computer-readable storage medium is provided, comprising:

[0080] When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the singing evaluation method as described in any of the preceding claims.

[0081] In a fourth aspect of this disclosure, a computing device is provided, comprising:

[0082] processor;

[0083] Memory used to store the processor's executable instructions;

[0084] The processor is configured to execute the executable instructions to implement the singing evaluation method as described in the preceding one.

[0085] According to the singing evaluation scheme provided in this disclosure, a multi-scale network model suitable for singing scenarios is constructed using deep learning technology. The convolutional layers in this multi-scale network model include convolutional kernels corresponding to multiple time scales. This multi-scale network model can achieve singing evaluation without reference information. Specifically, the multi-scale network model can perform feature processing on the spectrogram features of the input singing audio at different time scales, and exchange the processed information through feature fusion. This allows for the acquisition of useful information at different time scales, and ultimately, the probability distribution of different evaluation results is determined based on this useful information at different time scales. Attached Figure Description

[0086] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which:

[0087] Figure 1 schematically illustrates a singing scoring system provided in this disclosure;

[0088] Figure 2 schematically illustrates the singing evaluation method provided in this disclosure;

[0089] Figure 3 schematically illustrates a media diagram provided in this disclosure;

[0090] Figure 4 schematically illustrates a diagram of the singing evaluation device provided in this disclosure;

[0091] Figure 5 schematically illustrates a computing device provided in this disclosure.

[0092] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0093] The principles and spirit of this disclosure will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0094] Those skilled in the art will recognize that embodiments of this disclosure can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0095] According to embodiments of this disclosure, a singing evaluation method, a computer-readable storage medium, an apparatus, and a computing device are proposed.

[0096] In this article, it is important to understand that any number of elements in the accompanying figures is for illustrative purposes and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0097] The principles and spirit of this disclosure will be explained in detail below with reference to several representative embodiments.

[0098] The data disclosed herein may be data authorized by the user or fully authorized by all parties. The collection, dissemination, and use of the data shall comply with the requirements of relevant national laws and regulations. The implementation methods / executives of this disclosure may be combined with each other.

[0099] Invention Overview

[0100] This disclosure aims to provide a singing evaluation scheme that constructs a multi-scale network model suitable for singing scenarios using deep learning technology. The convolutional layers in this multi-scale network model include convolutional kernels corresponding to multiple time scales. This multi-scale network model can achieve singing evaluation without reference information. Specifically, the multi-scale network model can perform feature processing on the spectrogram features of the input singing audio at different time scales, and exchange the processed information through feature fusion. This allows for the acquisition of useful information at different time scales, and ultimately, the probability distribution of different evaluation results is determined based on this useful information.

[0101] After introducing the basic principles of this disclosure, various non-limiting embodiments of this disclosure will be described in detail below.

[0102] Application Scenarios Overview

[0103] Figure 1 illustrates a schematic diagram of an exemplary singing evaluation system architecture applicable to this paper. In Figure 1, various network nodes can communicate information via the network, thereby completing interaction and data processing. The singing evaluation system may include a server 105 that communicates with one or more clients 106 via network 112, and a database 115 that may be integrated into or independent of the server 105.

[0104] Each network 112 may include wired or wireless telecommunications devices, through which the network device on which client 106 is based may exchange data. For example, each network 112 may include a local area network (“LAN”), a wide area network (“WAN”), an intranet, the Internet, a mobile phone network, a virtual private network (VPN), a cellular or other mobile communication network, Bluetooth, NFC, or any combination thereof. In the discussion of exemplary embodiments, it should be understood that the terms “data” and “information” are used interchangeably herein to refer to text, images, audio, video, or any other form of information that may exist in a computer-based environment.

[0105] Each network device on which client 106 is based may include a device having a communication module capable of sending and receiving data via network 112. For example, each network device on which client 106 is based may include a server, desktop computer, laptop computer, tablet computer, smartphone, handheld computer, personal digital assistant (“PDA”), or any other wired or wireless processor-driven device. In the exemplary embodiment depicted in FIG1, the network device on which client 106 is based can be operated by a singing user.

[0106] Users (including individuals or organizations) can use applications such as web browser applications or standalone applications to view, download, upload, or otherwise access the singing software or web pages via the network.

[0107] A web browser application or a standalone application can interact with a web server (or other server, such as a singing platform, music platform, karaoke platform, etc.) connected to the network.

[0108] In Figure 1, the computing device 103, which can be integrated with or separate from the server 105, is typically connected via an internal or dedicated network, or an encrypted public network, in the latter case. In particular, when integrated, a more efficient and faster internal bus connection may be used. Whether integrated or separate, this computing device can access the database 115 directly or through the server 105.

[0109] By properly programming the computer device, the implementation of the methods described in this specification can be controlled through such instructions. In particular, when there is an integrated relationship, the transactions processed by the computer device can be regarded as the processing of the server 105 without special distinction.

[0110] In the aforementioned singing evaluation system, the client 106 can upload the acquired singing audio to the server 105, and the server 105 can process the singing audio to obtain an evaluation result for the singing audio.

[0111] The server can refer to a server, server cluster, or cloud platform built from a server cluster for singing software or applications.

[0112] In some embodiments, the singing audio can be processed locally on the client side of the singing software or application to obtain an evaluation result. When applied to the client side, this generally requires the client to have strong computing resources. The following embodiments will describe relevant implementations using the server side as the execution entity.

[0113] Exemplary methods

[0114] The method for evaluating singing according to an exemplary embodiment of this disclosure will now be described with reference to Figure 4 and the application scenario shown in Figure 1. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of this disclosure, and the embodiments of this disclosure are not limited in any way. Rather, the embodiments of this disclosure can be applied to any applicable scenario.

[0115] As shown in Figure 4, the singing evaluation method may include the following steps:

[0116] Step 210: Extract features from the acquired singing audio to obtain the spectral features of the singing audio.

[0117] In this specification, the singing audio may be an audio clip of a user humming, collected by a recording device (such as a microphone) from the surrounding environment; or an audio clip obtained from local or other devices or network storage (e.g., downloaded or cached).

[0118] In one exemplary embodiment, step 210 above may include:

[0119] The acquired singing audio is converted from the time domain to the frequency domain to obtain the spectral characteristics of the singing audio in the time-frequency domain; wherein, the spectral characteristics in the time-frequency domain include time information in the time domain and frequency information in the frequency domain.

[0120] Convolution calculations on singing audio in the time domain are complex and computationally intensive. Therefore, a time-to-frequency domain transformation is performed on the singing audio, converting it into spectral features in the time-frequency domain, which can then be used for convolution calculations in the frequency domain. Convolution calculations in the frequency domain are relatively simple, requiring only the spectral features to be multiplied by the convolution kernel.

[0121] In this specification, the above-mentioned time-domain to frequency-domain conversion processing can be performed using Fourier transform, short-time Fourier transform, CQT (Constant-Q transform), etc.

[0122] The following example uses CQT and short-time Fourier transform to illustrate the concept.

[0123] In one implementation, the step of performing a time-domain to frequency-domain conversion on the acquired singing audio to obtain the spectral characteristics of the singing audio in the time-frequency domain may include:

[0124] The acquired singing audio is subjected to constant Q transformation to obtain the spectrogram features of the singing audio in the time-frequency domain.

[0125] In another implementation, the step of performing a time-domain to frequency-domain conversion on the acquired singing audio to obtain the spectral characteristics of the singing audio in the time-frequency domain includes:

[0126] The acquired singing audio is processed by short-time Fourier transform, and the transformed features are converted into Mel-scale spectral features by Mel-scale filter banks.

[0127] In this manual, constant Q-transform and short-time Fourier transform are very important time-frequency analysis tools in audio processing technology, and are particularly suitable for music signal processing.

[0128] The spectrogram features obtained by constant Q transformation can also be called CQT features. Compared with other audio features, the Mel-scale spectrogram features obtained by Mel-scale transformation through Mel-scale filter are more consistent with human auditory perception of music. Therefore, using spectrogram features or Mel-scale features as input can make the evaluation results calculated by multi-scale network models more accurate, closer to or consistent with human perception of the quality of music.

[0129] Step 220: Input the spectral features into a multi-scale network model for calculation; wherein, the multi-scale network model includes a model based on Convolutional Neural Networks (CNN), and the convolutional layers in the multi-scale network model include convolutional kernels corresponding to multiple time scales.

[0130] Although the multi-scale network model in this specification uses the CNN network structure as the main framework of the model, in terms of the specific network structure, it does not stack a large number of convolutional layers as in existing CNN network models. Instead, it adopts a multi-timescale network structure, that is, it sets multiple convolutional kernels corresponding to different time scales in the convolutional layers.

[0131] After introducing the model structure of the multi-scale network model, the training process of the multi-scale network model will be further described below.

[0132] Similar to typical model training, the first step is to collect training samples for the model. It should be noted that the classification labels on these training samples include the labels for the different evaluation results.

[0133] In practical applications, at least three different evaluation results can be labeled. Taking three labels as an example, the training samples can be labeled as "good", "medium", and "poor" (this classification method can be called a three-class classification method).

[0134] In implementation, dry vocal clips (ranging from a few seconds to tens of seconds) of songs can be collected. Singing levels are then manually labeled using a method where one or more groups of experts with a certain level of musical literacy listen to and score the clips under identical conditions. The label for each clip is ultimately calculated using the experts' scores (e.g., by taking the mode, mean, or other methods) and converted into one of three categories: good, average, or poor. After accumulating a number of labeled clips (e.g., 1000 songs), this data can be used as training samples for model training.

[0135] Specifically, the training samples can be sequentially input into the multi-scale network model. When the multi-scale network model is trained based on the training samples, a multi-classification loss function (such as cross-entropy) can be used to calculate the loss, and backpropagation can be performed based on the loss result to update the model parameters. Through multiple iterations of training, relatively ideal model parameters can be obtained, and then the multi-scale network model can be used for singing evaluation.

[0136] The following section further describes the process by which the multi-scale network model calculates the input spectral features:

[0137] Step A1: The acoustic spectrum features are downsampled at multiple time scales along the time dimension to obtain acoustic spectrum sub-feature sequences corresponding to each time scale; wherein, the acoustic spectrum sub-feature sequence is composed of several acoustic spectrum sub-features obtained by downsampling at the same time scale in the order of sampling.

[0138] In an exemplary embodiment, downsampling the spectral features at multiple time scales along the time dimension may include:

[0139] For a spectral feature of size F*T, the spectral feature is downsampled at time intervals of T / 2, ..., T / (2^n) along the time dimension; where F is the frequency amplitude of the spectral feature, T is the duration of the spectral feature, and n is a positive integer. The size of n can be set according to the actual situation; the larger the n, the more spectral sub-features are downsampled.

[0140] This manual uses downsampling to reduce redundant information in the temporal spectrum features; thus, even with a smaller convolution kernel, convolution calculations can be performed on spectral features with a long temporal span (long duration), thereby obtaining longer contextual information.

[0141] In an exemplary embodiment, before downsampling the spectral features at the multiple time scales in the time dimension, the method further includes:

[0142] The spectral features are downsampled at various time scales along the frequency dimension.

[0143] In this specification, when downsampling in the time dimension, downsampling can also be performed in the frequency dimension (vertical direction); this can reduce the redundant information of the acoustic spectrum features in the frequency dimension at different time scales.

[0144] It is worth mentioning that downsampling also results in the loss of some detailed information in the spectral features. For example, highly correlated spectral features may be segmented, thus losing the effective information expressed by the correlation structure. Therefore, the convolutional layers of the multi-scale network model can retain the original time-scale convolutional kernels. In this way, the spectral feature sequence 1 corresponding to T / 2, ..., T / (2^n) obtained by downsampling, the spectral feature sequence 2 corresponding to T / 4 (not present when n=1), the spectral feature sequence 3 corresponding to T / 8 (not present when n=1 and 2), ..., the spectral feature sequence n corresponding to T / (2^n), and the original time-scale spectral features are added to the convolutional kernels of their respective time-scales for convolution calculation.

[0145] Step A2: Input the acoustic spectrum feature sequence into the convolution kernel of the corresponding time scale for convolution calculation to obtain the time scale feature output by the convolution kernel.

[0146] In one exemplary embodiment, the convolution kernels corresponding to the various time scales are of the same size.

[0147] It should be noted that having the same kernel size does not mean that the kernels are identical. By setting kernels of the same size, the compression ratio of spectral features at different time scales can be the same when they are convolved by their respective kernels, thus facilitating subsequent feature fusion.

[0148] Step A3: The time scale features of the multiple time scales are fused to obtain fused features.

[0149] In an exemplary embodiment, the step of fusing the time-scale features of the multiple time scales to obtain fused features includes:

[0150] Multiple time-scale features are upsampled in the time dimension to obtain a second time-scale feature with the same original time scale as the spectral feature;

[0151] Multiple second-time features are concatenated and then convolved using a convolution kernel for fusion to obtain fused features.

[0152] By using feature fusion, information can be exchanged between timescale features at different time scales, enabling multi-scale network models to obtain useful information from different time scales (e.g., long time scales can focus on global information, while short time scales can focus on local details). In this way, useful information can be comprehensively obtained from the spectral information (i.e., fused features), which in turn enables multi-classifiers to calculate more accurate evaluation results.

[0153] In one exemplary embodiment, after obtaining the fusion features, the process may further include:

[0154] Determine if the current loop count has reached the preset number of iterations;

[0155] If the preset number of iterations is not reached, increment the current loop count by 1, and use the fused feature as the new spectrogram feature. Then, re-execute the aforementioned step A1: perform downsampling of the spectrogram feature at the multiple time scales in the time dimension, until the current loop reaches the preset number of iterations.

[0156] By using multiple rounds of downsampling and feature fusion, useful information can be obtained more comprehensively from the acoustic spectrum information.

[0157] Step A4: Input the fused features into a multi-classifier for calculation to obtain the probability distribution of the different evaluation results.

[0158] The classifier can consist of another convolutional layer, a global average pooling (GAP) layer, and a fully connected layer; for example, a softmax multi-classifier.

[0159] Taking the softmax multi-classifier as an example, when calculating the probability distribution of different evaluation results, the softmax multi-classifier can calculate based on the input fusion feature and output the probability distribution of the probability value corresponding to the fusion feature under each evaluation result. These probability distributions can be output in vector form. For example, if the softmax multi-classifier has three evaluation results: "good," "medium," and "poor," then after inputting the fusion feature, it can output a vector [a, b, c] representing the probability value corresponding to each evaluation result; this vector represents the probability of "good" as a, "medium" as b, and "poor" as c.

[0160] It should be noted that the probability values ​​range from [0, 1], and the sum of all probability values ​​in the probability distribution is 1.

[0161] Step 230: Determine the final evaluation result based on the probability distribution of different evaluation results output by the multi-scale network model.

[0162] After obtaining the probability distributions of different evaluation results, it is still necessary to determine the final evaluation result from the multiple probability values ​​in the probability distribution.

[0163] The final evaluation result can be determined by taking the probability value of the maximum value.

[0164] The following example illustrates the different evaluation results, including at least three different evaluation results:

[0165] Step 230 may include:

[0166] Obtain the probability distributions of at least three different evaluation results output by the multi-scale network model, determine the largest probability value in the probability distribution, and determine the evaluation result corresponding to the largest probability value as the final evaluation result.

[0167] This instruction manual may also include, after determining the evaluation results of the singing audio, the following:

[0168] The scoring score of the singing audio is calculated based on the probability distribution of the different evaluation results.

[0169] In this way, by calculating the score, it is possible to compare the performance levels of singing audio with the same evaluation result.

[0170] In an exemplary embodiment, calculating the score of the singing audio based on the probability distribution of the different evaluation results includes:

[0171] Obtain the preset weight for each evaluation result;

[0172] The probability value of each scoring result in the probability distribution is multiplied by the corresponding preset weight, and the products of each scoring result are added together. The sum is determined as the score of the singing audio.

[0173] The following will still use three evaluation results as examples, and illustrate them with an exemplary scoring formula: P0*1.0+P1*0.5+P2*0.0.

[0174] Where P0, P1, and P2 are the probability values ​​corresponding to "good", "medium", and "poor" in the probability distribution, and 1.0, 0.5, and 0.0 are the preset weights corresponding to "good", "medium", and "poor".

[0175] This calculation formula can be used to calculate a specific score based on the probability distribution, and the score can be used to indicate the quality of the singing audio. Generally, the higher the score, the better the singing, and vice versa.

[0176] In addition, by setting a score threshold to determine whether the evaluation result is valid, and combining it with the calculated score, problems with abnormal evaluation results can be identified.

[0177] Specifically, in practical applications, when the probability values ​​of different evaluation results are not significantly different, the evaluation result determined in step 230 may be inaccurate. For example, suppose the probability values ​​corresponding to "good," "medium," and "poor" are 0.4, 0.3, and 0.3, respectively; then, since "good" has the highest probability value, the final evaluation result is "good." However, 0.4 is not actually a very high probability value, so this evaluation result is inappropriate. The score calculated using the above formula is: 0.4*1.0 + 0.3*0.5 + 0.3*0.0 = 0.55. Assuming the score threshold is 0.7, since 0.55 is less than 0.7, the evaluation result can be determined to be invalid, thus identifying inaccurate evaluation results.

[0178] In an exemplary embodiment, before inputting the spectral features into the multi-scale network model for computation, the method further includes:

[0179] The acoustic spectrum features are divided into several short acoustic spectrum features according to a preset duration;

[0180] The step of inputting the acoustic spectrum features into a multi-scale network model for calculation includes:

[0181] The short spectral features of the aforementioned segments are sequentially input into a multi-scale network model for calculation;

[0182] After calculating the score corresponding to each short spectral feature, the score corresponding to each short spectral feature is constructed into a score curve of the singing audio changing over time according to the chronological order of the short spectral features.

[0183] In this manual, when the input singing audio is long, it can be segmented for singing evaluation. A scoring curve can be constructed based on the obtained set of scores. The changes in the scoring curve can be used to visually show the changes in singing level during the singing process, thereby identifying the flaws or excellent moments in the singing process.

[0184] Exemplary media

[0185] After introducing the method of the exemplary embodiments of the present disclosure, the medium of the exemplary embodiments of the present disclosure will now be described with reference to FIG3.

[0186] In this exemplary embodiment, the above method can be implemented by a program product, such as a portable compact disc read-only memory (CD-ROM) containing program code, which can run on a device, such as a personal computer. However, the program product disclosed herein is not limited thereto. In this document, a readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0187] The program product may employ any combination of one or more readable media. A readable media may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0188] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0189] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RE, etc., or any suitable combination thereof.

[0190] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0191] In summary, this disclosure provides a computer-readable storage medium that, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the aforementioned singing evaluation method embodiments.

[0192] Exemplary device

[0193] Having introduced the medium of the exemplary embodiments of the present disclosure, the apparatus of the exemplary embodiments of the present disclosure will now be described with reference to FIG4.

[0194] Figure 4 schematically illustrates a block diagram of a singing evaluation device according to an embodiment of the present disclosure, corresponding to the method embodiment shown in Figure 2 above. The singing evaluation device may include:

[0195] Extraction unit 410 performs feature extraction on the acquired singing audio to obtain the spectral features of the singing audio;

[0196] Input unit 420 inputs the acoustic spectrum features into a multi-scale network model for calculation; wherein, the multi-scale network model includes a model based on a convolutional neural network, and the convolutional layers in the multi-scale network model include convolutional kernels corresponding to multiple time scales;

[0197] Unit 430 determines the final evaluation result based on the probability distribution of different evaluation results output by the multi-scale network model; wherein the calculation process of the multi-scale network model includes the following units:

[0198] Sampling unit 421 downsamples the acoustic spectrum features at multiple time scales in the time dimension to obtain acoustic spectrum sub-feature sequences corresponding to each time scale; wherein, the acoustic spectrum sub-feature sequence is composed of several acoustic spectrum sub-features obtained by downsampling at the same time scale in the order of sampling.

[0199] The calculation unit 423 inputs the acoustic spectrum feature sequence into the convolution kernel of the corresponding time scale for convolution calculation to obtain the time scale feature output by the convolution kernel;

[0200] The fusion unit 425 fuses the time scale features of the multiple time scales to obtain fused features;

[0201] Classification unit 429 inputs the fused features into a multi-classifier for calculation to obtain the probability distribution of the different evaluation results.

[0202] Optionally, the extraction unit 410 is further configured to perform time-domain to frequency-domain conversion processing on the acquired singing audio to obtain the spectral characteristics of the singing audio in the time-frequency domain; wherein, the spectral characteristics in the time-frequency domain include time information in the time domain and frequency information in the frequency domain.

[0203] Optionally, the extraction unit 410 further includes:

[0204] The first processing subunit 411 performs constant Q-transformation on the acquired singing audio to obtain the spectrogram features of the singing audio in the time-frequency domain;

[0205] Alternatively, the second processing subunit 412 performs a short-time Fourier transform on the acquired singing audio and converts the transformed features into Mel-scale spectral features using a Mel-scale filter.

[0206] Optionally, the sampling unit 421 is further configured to downsample the spectral feature of size F*T in the time dimension at time intervals of T / 2, ..., T / (2^n); where F is the frequency amplitude of the spectral feature, T is the duration of the spectral feature, and n is a positive integer.

[0207] Optionally, the convolution kernels corresponding to the various time scales are of the same size.

[0208] Optionally, the sampling unit 421 is further configured to downsample the acoustic spectral features at the multiple time scales in the frequency dimension.

[0209] Optionally, the fusion unit 425 further includes:

[0210] The first fusion subunit 426 upsamples multiple time-scale features in the time dimension to obtain a second time-scale feature that is the same as the original time-scale of the sound spectrum feature.

[0211] The second fusion subunit 427 connects multiple second time features and performs convolution calculations using a convolution kernel for fusion to obtain fused features.

[0212] Optionally, following the fusion unit 425, the following may also be included:

[0213] The loop unit 428 determines whether the current loop count has reached the preset number of times; if the preset number of times has not been reached, the current loop count is incremented by 1, and the fused feature is used as the new spectral feature. The sampling unit is then re-executed until the current loop reaches the preset number of times.

[0214] Optional, also includes:

[0215] The scoring unit 440 calculates the score of the singing audio based on the probability distribution of the different evaluation results.

[0216] Optionally, the scoring unit 440 further includes:

[0217] Obtain subunit 442 and obtain the preset weight for each evaluation result;

[0218] In molecule unit 444, the probability value of each scoring result in the probability distribution is multiplied by the corresponding preset weight, and the product results of each scoring result are added together. The sum is determined as the score of the singing audio.

[0219] Optionally, prior to the input unit 420, the following may also be included:

[0220] The segmentation unit 413 divides the acoustic spectrum features into several short acoustic spectrum features according to a preset duration;

[0221] The input unit 420 is further used to sequentially input the short spectral features of the several segments into the multi-scale network model for calculation;

[0222] Following the scoring unit 440, the following is also included:

[0223] After calculating the score corresponding to each short spectral feature, the construction unit 446 constructs a score curve of the singing audio changing over time according to the time sequence of the short spectral features.

[0224] Optionally, the classification labels of the training samples of the multi-scale network model include the labels of the different evaluation results;

[0225] When training the model based on the training samples, the multi-scale network model uses a multi-classification loss function to calculate the loss and performs backpropagation based on the loss results to update the model parameters.

[0226] Optionally, the different evaluation results include at least three different evaluation results;

[0227] The determining unit 430 is further configured to obtain the probability distribution of at least three different evaluation results output by the multi-scale network model, determine the largest probability value in the probability distribution, and determine the evaluation result corresponding to the largest probability value as the final evaluation result.

[0228] Exemplary computing device

[0229] Having introduced the methods, media, and apparatus of exemplary embodiments of the present disclosure, the computing device of exemplary embodiments of the present disclosure will now be described with reference to FIG5.

[0230] The computing device 1500 shown in Figure 5 is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0231] As shown in Figure 5, the computing device 1500 is presented in the form of a general-purpose computing device. The components of the computing device 1500 may include, but are not limited to: at least one processing unit 1501, at least one storage unit 1502, and a bus 1503 connecting different system components (including the processing unit 1501 and the storage unit 1502).

[0232] Bus 1503 includes a data bus, a control bus, and an address bus.

[0233] Storage unit 1502 may include readable media in the form of volatile memory, such as random access memory (RAM) 15021 and / or cache memory 15022, and may further include readable media in the form of non-volatile memory, such as read-only memory (ROM) 15023.

[0234] Storage unit 1502 may also include a program / utility 15025 having a set (at least one) of program modules 15024, such program modules 15024 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0235] The computing device 1500 can also communicate with one or more external devices 1504 (such as a keyboard, pointing device, etc.).

[0236] This communication can be performed via input / output (I / O) interface 1505. Furthermore, computing device 1500 can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 1506. As shown in Figure 5, network adapter 1506 communicates with other modules of computing device 1500 via bus 1503. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with computing device 1500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0237] The aforementioned singing evaluation method can be implemented using the computing device 1500 shown in Figure 5.

[0238] It should be noted that although several units / modules or sub-units / modules of the singing evaluation device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0239] Furthermore, although the operations of the methods disclosed herein are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0240] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A singing evaluation method, comprising: Feature extraction is performed on the acquired singing audio to obtain the spectral features of the singing audio; The spectral features are input into a multi-scale network model for calculation; wherein the multi-scale network model includes a convolutional neural network-based model, and the convolutional layers in the multi-scale network model include convolutional kernels corresponding to multiple time scales; the final evaluation result is determined based on the probability distribution of different evaluation results output by the multi-scale network model; wherein the calculation process of the multi-scale network model includes: downsampling the spectral features at multiple time scales in the time dimension to obtain spectral sub-feature sequences corresponding to each time scale; wherein the spectral sub-feature sequences are composed of several spectral sub-features downsampled at the same time scale in the sampling order; inputting the spectral sub-feature sequences into the convolutional kernels of the corresponding time scales for convolution calculation to obtain the time scale features output by the convolutional kernels; fusing the time scale features of the multiple time scales to obtain fused features; inputting the fused features into a multi-classifier for calculation to obtain the probability distribution of the different evaluation results.

2. The method according to claim 1, wherein the step of extracting features from the acquired singing audio to obtain the spectrogram features of the singing audio includes: The acquired singing audio is converted from the time domain to the frequency domain to obtain the spectral characteristics of the singing audio in the time-frequency domain; wherein, the spectral characteristics in the time-frequency domain include time information in the time domain and frequency information in the frequency domain.

3. The method according to claim 2, wherein performing time-domain to frequency-domain conversion on the acquired singing audio to obtain the spectral characteristics of the singing audio in the time-frequency domain includes: The acquired singing audio is subjected to constant Q-transformation to obtain the spectrogram features of the singing audio in the time-frequency domain; or, the acquired singing audio is subjected to short-time Fourier transform, and the transformed features are converted to Mel-scale spectrogram features through a Mel-scale filter.

4. The method according to claim 1, wherein downsampling the spectral features at multiple time scales in the time dimension comprises: For a spectral feature of size F*T, the spectral feature is downsampled in the time dimension at time intervals of T / 2, ..., T / (2^n); where F is the frequency amplitude of the spectral feature, T is the duration of the spectral feature, and n is a positive integer.

5. The method according to claim 1, wherein the convolution kernels corresponding to the multiple time scales are of the same size.

6. The method according to claim 1, further comprising, before downsampling the spectral features at the multiple time scales in the time dimension: The spectral features are downsampled at various time scales along the frequency dimension.

7. The method according to claim 1, wherein fusing the time-scale features of the multiple time scales to obtain fused features includes: Multiple time-scale features are upsampled in the time dimension to obtain a second time-scale feature with the same original time scale as the spectral feature; Multiple second-time features are concatenated and then convolved using a convolution kernel for fusion to obtain fused features.

8. The method according to claim 1, further comprising, after obtaining the fusion features: Determine if the current loop count has reached the preset number of iterations; If the preset number of iterations is not reached, the current loop count is incremented by 1, and the fused feature is used as the new spectrogram feature. The process of downsampling the spectrogram feature at multiple time scales in the time dimension is repeated until the current loop reaches the preset number of iterations.

9. The method according to claim 1, further comprising: The scoring score of the singing audio is calculated based on the probability distribution of the different evaluation results.

10. The method according to claim 9, wherein calculating the score of the singing audio based on the probability distribution of the different evaluation results includes: Obtain the preset weight for each evaluation result; The probability value of each scoring result in the probability distribution is multiplied by the corresponding preset weight, and the products of each scoring result are added together. The sum is determined as the score of the singing audio.

11. The method according to claim 9, further comprising, before inputting the spectral features into a multi-scale network model for calculation: The acoustic spectrum features are divided into several short acoustic spectrum features according to a preset duration; The step of inputting the acoustic spectrum features into a multi-scale network model for calculation includes: sequentially inputting the short acoustic spectrum features of the several segments into the multi-scale network model for calculation; After calculating the score corresponding to each short spectral feature, the score corresponding to each short spectral feature is constructed into a score curve of the singing audio changing over time according to the chronological order of the short spectral features.

12. The method according to claim 1, wherein the classification labels of the training samples of the multi-scale network model include the labels of the different evaluation results; when the multi-scale network model is trained based on the training samples, it uses a multi-classification loss function to calculate the loss and performs backpropagation based on the loss results to update the model parameters.

13. The method according to claim 1, wherein the different evaluation results include at least three different evaluation results; and the determination of the final evaluation result based on the probability distribution of the different evaluation results output by the multi-scale network model includes: Obtain the probability distributions of at least three different evaluation results output by the multi-scale network model, determine the largest probability value in the probability distribution, and determine the evaluation result corresponding to the largest probability value as the final evaluation result.

14. A singing evaluation device, the device comprising: The extraction unit performs feature extraction on the acquired singing audio to obtain the spectral features of the singing audio; The system comprises: an input unit that inputs the spectral features into a multi-scale network model for calculation; wherein the multi-scale network model includes a convolutional neural network-based model, and the convolutional layers in the multi-scale network model include convolutional kernels corresponding to multiple time scales; a determination unit that determines the final evaluation result based on the probability distribution of different evaluation results output by the multi-scale network model; wherein the calculation process of the multi-scale network model includes the following units: a sampling unit that downsamples the spectral features at multiple time scales along the time dimension to obtain spectral sub-feature sequences corresponding to each time scale; wherein the spectral sub-feature sequences are composed of several spectral sub-features downsampled at the same time scale in the order of sampling; a calculation unit that inputs the spectral sub-feature sequences into the convolutional kernels at the corresponding time scales for convolutional calculation to obtain the time scale features output by the convolutional kernels; a fusion unit that fuses the time scale features at multiple time scales to obtain fused features; and a classification unit that inputs the fused features into a multi-classifier for calculation to obtain the probability distribution of the different evaluation results.

15. The apparatus according to claim 14, wherein the extraction unit is further configured to perform time-domain to frequency-domain conversion processing on the acquired singing audio to obtain the spectral characteristics of the singing audio in the time-frequency domain; wherein, The acoustic spectrum features in the time-frequency domain include time information in the time domain and frequency information in the frequency domain.

16. The apparatus according to claim 15, wherein the extraction unit further comprises: The first processing subunit performs a constant Q-transform on the acquired singing audio to obtain the spectrogram features of the singing audio in the time-frequency domain; or, the second processing subunit performs a short-time Fourier transform on the acquired singing audio and converts the transformed features to Mel-scale spectrogram features through a Mel-scale filter.

17. The apparatus according to claim 14, wherein the sampling unit is further configured to downsample the spectral features of size F*T in the time dimension for time intervals of T / 2, ..., T / (2^n); wherein, F is the frequency amplitude of the spectral feature, T is the duration of the spectral feature, and n is a positive integer.

18. The apparatus of claim 14, wherein the convolution kernels corresponding to the multiple time scales are of the same size.

19. The apparatus according to claim 14, wherein the sampling unit is further configured to downsample the acoustic spectral features at the multiple time scales in the frequency dimension.

20. The apparatus of claim 14, wherein the fusion unit further comprises: The first fusion subunit upsamples multiple time-scale features in the time dimension to obtain a second time-scale feature that is the same as the original time-scale of the acoustic spectrum feature. The second fusion subunit connects multiple second temporal features and performs convolution calculations using a convolution kernel for fusion to obtain fused features.

21. The apparatus of claim 14, further comprising, after the fusion unit: The loop unit determines whether the current loop count has reached the preset count. If the preset number of iterations is not reached, the current loop count is incremented by 1, and the fused feature is used as the new spectral feature. The sampling unit is then re-executed until the current loop reaches the preset number of iterations.

22. The apparatus of claim 14, further comprising: The scoring unit calculates the score of the singing audio based on the probability distribution of the different evaluation results.

23. The apparatus according to claim 22, wherein the scoring unit further comprises: Obtain the sub-unit and the preset weight for each evaluation result; The process involves multiplying the probability value of each scoring result in the probability distribution by its corresponding preset weight, summing the products of each scoring result, and determining the summed result as the score of the singing audio.

24. The apparatus of claim 22, further comprising, before the input unit: The segmentation unit divides the spectral features into several short spectral features according to a preset duration; The input unit is further configured to sequentially input the short spectral features of the plurality of segments into a multi-scale network model for calculation; after the scoring unit, it further includes a construction unit, which, after calculating the score value corresponding to each short spectral feature, constructs the score value corresponding to each short spectral feature into a score value curve of the singing audio changing over time according to the temporal order of the short spectral features of the plurality of segments.

25. The apparatus according to claim 14, wherein the classification labels of the training samples of the multi-scale network model include the labels of the different evaluation results; when the multi-scale network model is trained based on the training samples, it uses a multi-classification loss function to calculate the loss and performs backpropagation based on the loss results to update the model parameters.

26. The apparatus according to claim 14, wherein the different evaluation results include at least three different evaluation results; the determining unit is further configured to obtain the probability distribution of the at least three different evaluation results output by the multi-scale network model, determine the largest probability value in the probability distribution, and determine the evaluation result corresponding to the largest probability value as the final evaluation result.

27. A computer-readable storage medium comprising: When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the singing evaluation method as described in any one of claims 1-13.

28. A computing device, comprising: processor; A memory for storing processor-executable instructions; wherein the processor is configured to execute the executable instructions to implement the singing evaluation method as described in any one of claims 1-13.

Images