Vocal music training intelligent evaluation method and system based on voice analysis

Abstract

The invention relates to the technical field of vocal music training evaluation, and discloses a vocal music training intelligent evaluation method and system based on voice analysis. The method comprises the following steps: firstly, acquiring vocal music training original data, and processing and aligning multi-modal data by using a transfer learning small sample completion method when the data is insufficient; constructing a pronunciation stability evaluation model and a comprehensive pronunciation performance evaluation model; constructing a pronunciation scene simulator for different vocal music styles; the evaluation parameters are analyzed through a Shapley value decomposition algorithm, and a lightweight index system is constructed; and establishing a training scheme optimization agent model by using Gaussian process regression and a Bayesian optimization algorithm. The system comprises a data acquisition and completion module, an evaluation module, a simulation module, an analysis module, an optimization module and the like. The vocal music training system can comprehensively and accurately evaluate vocal music training, adapts to various styles, optimizes a training scheme, and assists a singer to improve the singing level through real-time feedback and visual display.

Description

Technical Field

[0001] The present invention relates to the technical field of vocal training assessment, and in particular to a vocal training intelligent assessment method and system based on speech analysis. Background Art

[0002] In the field of vocal training, traditional evaluation methods mainly rely on the subjective judgment of professional teachers. This method has many limitations: on the one hand, teachers' personal experience and aesthetic differences can lead to inconsistent evaluation standards. For example, different teachers may have very different evaluations of the same singer's pitch and timbre, making it difficult for students to obtain objective and accurate feedback, which is not conducive to students formulating targeted training plans. On the other hand, manual evaluation is inefficient. When faced with a large number of students, it is difficult for teachers to track and analyze the training process of each student in a detailed and comprehensive manner.

[0003] With the development of science and technology, some vocal training auxiliary tools have begun to appear, such as some simple pitch monitoring software. However, these tools have relatively simple functions and can only perform basic measurement and display of pitch, and cannot comprehensively evaluate multiple key elements in vocal training. Vocal training involves multimodal data such as audio signals, pitch sequences, rhythmic characteristics, and pronunciation physiological signals. Single pitch monitoring cannot cover these complex data information and it is difficult to fully reflect the singer's vocal condition.

[0004] At the same time, existing technologies are not capable of processing small sample data. In the actual vocal training data collection process, the amount of training data obtained is often insufficient due to various conditions, such as the limitations of the collection equipment and the difficulty in collecting samples due to the large individual differences among singers. Insufficient data will seriously affect the accuracy and generalization ability of the evaluation model, causing the model to perform poorly when faced with new singing samples.

[0005] In addition, different vocal styles (such as bel canto, pop, and folk singing) have their own unique pronunciation characteristics and technical requirements. Most of the existing evaluation methods and systems are unable to effectively distinguish and adapt to different styles, and it is difficult to meet the diverse needs of vocal training. For example, bel canto focuses on the resonance of the voice and the control of breath, while pop singing emphasizes emotional expression and personalized voice characteristics, and existing evaluation methods cannot accurately evaluate and guide these differences.

[0006] Moreover, the existing technology fails to fully integrate physiological sensor data for the evaluation of pronunciation effectiveness. Physiological data such as breathing rate, laryngeal muscle electrical signals and chest pressure are of great significance for evaluating the stability and effectiveness of pronunciation, but traditional evaluation methods ignore the value of these data, resulting in incomplete and in-depth evaluation results, and unable to provide singers with effective suggestions for optimizing their voice from a physiological level. Summary of the invention

[0007] The purpose of the present invention is to provide an intelligent evaluation method and system for vocal music training based on speech analysis to solve the problems raised in the above-mentioned background technology.

[0008] To achieve the above object, the present invention provides the following technical solution: an intelligent evaluation method for vocal music training based on speech analysis, the method comprising:

[0009] Step 1: Obtain the original data of vocal music training, including audio signals, pitch sequences, rhythm features, and pronunciation physiological signals; when the data volume is insufficient, a small-sample completion method based on transfer learning is adopted, the potential features are extracted by a pre-trained speech encoder, and the variational autoencoder is combined to generate and complete the missing rhythm features and timbre parameters, and the multi-modal data is aligned by the dynamic time warping algorithm;

[0010] Step 2: Based on the completed multi-modal data, construct a pronunciation stability evaluation model; with the fundamental frequency trajectory, formant distribution, and glottal closure rate as the core features, use a dynamic time convolutional network to capture temporal dependencies, combine the attention mechanism to quantitatively model the pronunciation coherence, and introduce an adversarial training strategy to improve the robustness of the model to noise;

[0011] Step 3: Integrate the physiological sensor data to establish a comprehensive pronunciation efficiency evaluation model; through the synchronously collected respiratory frequency, laryngeal muscle electrical signals, and thoracic pressure data, use the multi-source heterogeneous data fusion technology to construct a dynamic relationship model based on a graph neural network to quantify the correlation weight between respiratory support and pitch stability;

[0012] Step 4: For different vocal music styles, construct a pronunciation scenario simulator; use a conditional adversarial generation network to generate a synthetic speech data set including bel canto, pop, and ethnic singing methods, and align the feature distributions of real recordings and synthetic data through domain adaptation methods to enhance the cross-style generalization ability of the model;

[0013] Step 5: Conduct multi-dimensional sensitivity analysis on the evaluation parameters; use the Shapley value decomposition algorithm to identify the influence weights of pitch deviation, breath control, and resonance efficiency on the comprehensive score, screen the core evaluation indicators based on the feature importance ranking, and construct a lightweight evaluation index system;

[0014] Step 6: Establish an optimization proxy model for the vocal music training plan; use Gaussian process regression to construct a pronunciation improvement path prediction model, and combine the Bayesian optimization algorithm to hierarchically optimize the training intensity, practice duration, and vocalization position parameters, giving priority to optimizing the short-term modifiable parameters, and then iteratively adjusting the long-term training goals.

[0015] Preferably, in the step 1, the small-sample completion method of transfer learning specifically includes: using a pre-trained Wav2Vec 2.0 model to extract the context embedding vector of the speech signal, mapping the missing rhythm features to the latent space, generating the completed data through a VAE decoder, and using a contrastive learning loss function to constrain the distribution consistency between the generated data and the real data.

[0016] Preferably, in the step 2, the dilated convolution structure is adopted in the dynamic time convolutional network, the dilation coefficient increases exponentially, a gating mechanism is embedded between network layers, the output layer classifies the pronunciation coherence into five levels through the Softmax function, and the Focal Loss is used to alleviate the class imbalance problem.

[0017] Preferably, in the step 3, the nodes of the graph neural network include the breathing frequency, the peak value of the laryngeal electromyogram signal, and the pitch variance, the edge weights are calculated by the mutual information algorithm, the model updates the node states by using a message passing mechanism, and outputs the comprehensive pronunciation efficiency score through a graph pooling operation.

[0018] Preferably, in the step 4, the generator of the conditional adversarial generation network adopts a U-Net structure, the discriminator introduces spectral normalization constraints, the conditional vector includes the singing method type label and the singer's vocal range characteristics, and the loss function adopts a combination of the Wasserstein distance and the gradient penalty term.

[0019] Preferably, in the step 5, the Shapley value decomposition algorithm approximates the marginal contribution through Monte Carlo sampling, performs local interpretability analysis on the non-linear evaluation model, and uses the hierarchical clustering algorithm to group and reduce the dimensions of the highly correlated parameters.

[0020] Preferably, in the step 6, the kernel function of the Gaussian process regression adopts a combination of the Matern 5 / 2 kernel and the periodic kernel, the hyperparameters are optimized by maximum likelihood estimation, the acquisition function of the Bayesian optimization adopts an improved upper confidence bound strategy, and a dynamic balance coefficient is introduced to adjust the weights of exploration and exploitation.

[0021] Preferably, the method further includes a real-time feedback module, specifically: updating the evaluation model parameters through online incremental learning, processing the real-time audio stream by using a streaming computing framework, and displaying the pronunciation weakness distribution through the three-dimensional sound field visualization technology.

[0022] Preferably, the three-dimensional sound field visualization technology decomposes the sound wave propagation direction by using spherical harmonic functions, generates a virtual auditory scene in combination with the head-related transfer function, and superimposes the weakness area on the vocal tract anatomical model through a heat map.

[0023] Preferably, the present invention further includes a vocal music training intelligent evaluation system based on speech analysis, and the system includes:

[0024] Data Acquisition and Completion Module: It is used to acquire the original data of vocal music training, including audio signals, pitch sequences, rhythm features, and pronunciation physiological signals. When the amount of data is insufficient, a small-sample completion method based on transfer learning is adopted. The pre-trained speech encoder is used to extract latent features, and the variational autoencoder is combined to generate and complete the missing rhythm features and timbre parameters, and the dynamic time warping algorithm is used to align the multi-modal data;

[0025] Pronunciation Stability Evaluation Module: Based on the completed multi-modal data, with the fundamental frequency trajectory, formant distribution, and glottal closure rate as the core features, the dynamic time convolutional network is used to capture the temporal dependence, and the attention mechanism is combined to quantitatively model the pronunciation coherence, and the adversarial training strategy is introduced to improve the robustness of the model to noise, and a pronunciation stability evaluation model is constructed;

[0026] Comprehensive Pronunciation Efficiency Evaluation Module: It is used to fuse the physiological sensor data. Through the synchronously collected respiratory frequency, laryngeal muscle electrical signal, and thoracic pressure data, the multi-source heterogeneous data fusion technology is adopted to construct a dynamic relationship model based on the graph neural network, quantify the correlation weight between respiratory support and pitch stability, and establish a comprehensive pronunciation efficiency evaluation model;

[0027] Pronunciation Scenario Simulation Module: For different vocal music styles, the conditional adversarial generation network is used to generate a synthetic speech data set including bel canto, pop, and ethnic singing methods, and the domain adaptation method is used to align the feature distributions of real recordings and synthetic data, enhance the cross-style generalization ability of the model, and construct a pronunciation scenario simulator;

[0028] Evaluation Parameter Analysis Module: Perform multi-dimensional sensitivity analysis on the evaluation parameters, use the Shapley value decomposition algorithm to identify the influence weights of pitch deviation, breath control, and resonance efficiency on the comprehensive score, screen the core evaluation indicators based on the feature importance ranking, and construct a lightweight evaluation index system;

[0029] Training Plan Optimization Module: Use Gaussian process regression to construct a pronunciation improvement path prediction model, and combine the Bayesian optimization algorithm to hierarchically optimize the training intensity, practice duration, and vocalization position parameters. First, optimize the short-term modifiable parameters, and then iteratively adjust the long-term training goals to establish a vocal music training plan optimization proxy model.

[0030] Compared with the prior art, the beneficial effects of the present invention are:

[0031] The present invention adopts a few-shot completion method based on transfer learning. When the amount of original vocal training data is insufficient, a pre-trained speech encoder (such as the Wav2Vec 2.0 model) is used to extract latent features, and a variational autoencoder is combined to generate and complete the missing rhythm features and timbre parameters. This method can effectively solve the problem of insufficient data volume, ensure the integrity of the data, and provide a reliable data basis for subsequent evaluation. Compared with traditional methods, it avoids evaluation biases caused by data missing and improves the accuracy of evaluation. For example, in some niche vocal styles or specific training scenarios where data collection is difficult, this method can effectively complete the data with limited data, enabling the evaluation model to learn more comprehensive features.

[0032] The constructed pronunciation stability evaluation model takes the fundamental frequency trajectory, formant distribution, and glottal closure rate as core features, and uses a dynamic time convolutional network to capture temporal dependencies. The use of the dilated convolution structure and the gating mechanism enables the network to capture the temporal features of speech signals at different scales, avoiding the problems of gradient vanishing or explosion, and enhancing the ability to capture important information. By combining the attention mechanism to quantitatively model pronunciation coherence, it can more accurately evaluate the pronunciation stability of singers. The introduction of the adversarial training strategy improves the robustness of the model to noise, and even in a training environment with external interference, it can accurately evaluate the pronunciation situation and provide stable and reliable evaluation results for singers.

[0033] A comprehensive pronunciation efficiency evaluation model is established by integrating physiological sensor data. By synchronously collecting respiratory frequency, laryngeal muscle electrical signal, and thoracic pressure data, a multi-source heterogeneous data fusion technology and a dynamic relationship model based on graph neural network are used to quantify the correlation weight between respiratory support and pitch stability. This method deeply analyzes pronunciation efficiency from a physiological level, comprehensively reflects the vocalization status of singers, and provides more targeted training suggestions for singers. For example, by analyzing the relationship between respiratory frequency and pitch stability, singers can understand how to adjust their breathing to improve pitch control and singing level.

[0034] A pronunciation scenario simulator is constructed for different vocal styles. A conditional adversarial generation network is used to generate a synthetic speech dataset including bel canto, pop, and ethnic singing styles, and the domain adaptation method is used to align the feature distributions of real recordings and synthetic data. This enables the model to have strong cross-style generalization ability and can adapt to the training and evaluation of multiple vocal styles. Whether it is the strict normative requirements of bel canto, the personalized expression of pop singing, or the unique charm of ethnic singing, the present invention can accurately evaluate and meet the diverse vocal training needs, providing professional evaluation services for singers of different styles.

[0035] Perform multi-dimensional sensitivity analysis on the evaluation parameters, use the Shapley value decomposition algorithm to identify the influence weights of pitch deviation, breath control, and resonance efficiency on the comprehensive score, screen the core evaluation indicators based on the feature importance ranking, and construct a lightweight evaluation index system. This not only improves the evaluation efficiency and reduces the consumption of computing resources, but also highlights the key evaluation factors, making the evaluation results more targeted. At the same time, use Gaussian process regression to construct a pronunciation improvement path prediction model, and combine the Bayesian optimization algorithm to hierarchically optimize the training intensity, practice duration, and vocalization position parameters. Prioritize optimizing the short-term modifiable parameters, and then iteratively adjust the long-term training goals to formulate a scientific and reasonable training plan for the singer, improve the training effect, and help the singer improve the singing level faster. Brief Description of the Drawings

[0036] Figure 1 It is the working principle diagram of the intelligent evaluation method for vocal music training based on speech analysis described in the present invention;

[0037] Figure 2 It is the working flow chart of the transfer learning small sample completion method;

[0038] Figure 3 It is the construction flow chart of the pronunciation stability evaluation model;

[0039] Figure 4 It is the construction flow chart of the pronunciation scene simulator. Detailed Embodiments

[0040] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0041] Please refer to Figures 1-4 , the present invention provides a technical solution: an intelligent evaluation method for vocal music training based on speech analysis, the method includes:

[0042] Step 1, obtain and complete the original data of vocal music training: Obtain the original data of vocal music training, including audio signals, pitch sequences, rhythm features, and pronunciation physiological signals. If the data volume is insufficient, use the small sample completion method based on transfer learning, extract potential features using a pre-trained speech encoder, combine the variational autoencoder to generate and complete the missing rhythm features and timbre parameters, and then align the multi-modal data through the dynamic time warping algorithm.

[0043] Step 2: Construct a pronunciation stability evaluation model: Based on the complemented multi-modal data, with the fundamental frequency trajectory, formant distribution, and glottal closure rate as the core features, use a dynamic time convolutional network to capture temporal dependencies, combine the attention mechanism to quantitatively model pronunciation coherence, and introduce an adversarial training strategy to improve the model's robustness to noise.

[0044] Step 3: Establish a comprehensive pronunciation efficiency evaluation model: Integrate physiological sensor data, and through synchronously collected respiratory frequency, laryngeal muscle electrical signal, and thoracic pressure data, use multi-source heterogeneous data fusion technology to construct a dynamic relationship model based on a graph neural network to quantify the correlation weight between respiratory support and pitch stability.

[0045] Step 4: Construct a pronunciation scenario simulator: For different vocal music styles, use a conditional adversarial generation network to generate a synthetic speech dataset containing bel canto, pop, and ethnic singing styles, and align the feature distributions of real recordings and synthetic data through domain adaptation methods to enhance the model's cross-style generalization ability.

[0046] Step 5: Conduct multi-dimensional sensitivity analysis on evaluation parameters: Use the Shapley value decomposition algorithm to identify the influence weights of pitch deviation, breath control, and resonance efficiency on the comprehensive score, screen core evaluation indicators based on feature importance ranking, and construct a lightweight evaluation index system.

[0047] Step 6: Establish an optimization proxy model for vocal music training programs: Use Gaussian process regression to construct a pronunciation improvement path prediction model, and combine the Bayesian optimization algorithm to hierarchically optimize training intensity, practice duration, and vocalization position parameters, giving priority to optimizing short-term modifiable parameters, and then iteratively adjusting long-term training goals.

[0048] The following further illustrates the present invention in conjunction with Embodiments 1 to 5:

[0049] Embodiment 1:

[0050] This embodiment elaborates in detail how to process the original data of vocal music training using the small-sample complement method of transfer learning when the data volume is insufficient, ensuring the integrity and accuracy of the data, and providing a reliable data basis for subsequent model training and evaluation. The specific steps include:

[0051] In Step 1, the specific operation of the small-sample complement method of transfer learning is as follows: Use a pre-trained Wav2Vec2.0 model to extract the context embedding vectors of the speech signal. The Wav2Vec 2.0 model is a model pre-trained on large-scale speech data, which can learn rich feature representations of speech signals. Through this model, the input speech signal can be converted into embedding vectors with context information, and these vectors contain important information such as the prosody and timbre of the speech.

[0052] After mapping the missing rhythm features to the latent space, the VAE (Variational Autoencoder) decoder is used to generate the completed data. VAE is a generative model composed of an encoder and a decoder. In this embodiment, the encoder maps the data with missing rhythm features to the latent space, and the decoder samples from the latent space and generates the completed data. To ensure the distribution consistency between the generated data and the real data, a contrastive learning loss function is used for constraint. The formula of the contrastive learning loss function is:

[0053]

[0054] where N represents the number of samples, z i is the feature representation of the real data, is the positive sample feature representation corresponding to z i , and sim represents the similarity function for calculating the similarity between two feature vectors. By minimizing this loss function, the generated data is made closer to the real data in terms of feature distribution.

[0055] In practical applications, assume there is a set of vocal music training data with some missing rhythm features. First, this set of data is input into a pre-trained Wav2Vec 2.0 model to obtain the context embedding vectors of the speech signals. Then, the data part with missing rhythm features is input into the encoder of the VAE and mapped to the latent space. Next, the decoder of the VAE is used to generate the completed rhythm feature data from the latent space. During the generation process, the parameters of the decoder are continuously adjusted to minimize the contrastive learning loss function between the generated data and the real data. After multiple iterations, the finally obtained completed rhythm feature data, together with other modality data (such as audio signals, pitch sequences, etc.) in the original data, provides complete data support for subsequent model training.

[0056] Embodiment 2:

[0057] In step 2, the dilated convolution structure is adopted in the dynamic time convolutional network. Dilated convolution is a special convolution operation that increases the receptive field of the convolutional kernel by inserting intervals between the elements of the convolutional kernel. In this embodiment, the dilation coefficient increases exponentially, which enables the network to capture the temporal features of the speech signal at different scales. For example, the dilation coefficient of the initial layer can be set to 1, and the dilation coefficients of the subsequent layers are set to 2, 4, 8, etc. in sequence.

[0058] A gating mechanism is embedded between network layers. The gating mechanism is similar to a switch that can decide whether to let information pass according to the features of the input signal. In the dynamic time convolutional network, the gating mechanism can effectively control the information flow, avoid the problems of gradient vanishing or gradient explosion, and at the same time enhance the network's ability to capture important information.

[0059] The output layer classifies the pronunciation coherence into five levels through the Softmax function. The formula of the Softmax function is:

[0060]

[0061] where \(x\) i is the \(i\)-th element in the input vector, and \(K\) is the number of classification categories. In this embodiment, \(K = 5\), that is, the pronunciation coherence is divided into five levels, such as very coherent, relatively coherent, average, relatively incoherent, and very incoherent. Through the Softmax function, the output of the network is converted into the probability values corresponding to each category, so as to realize the classification of pronunciation coherence.

[0062] To alleviate the class imbalance problem, Focal Loss is adopted. The class imbalance problem refers to the situation where the number of samples in different categories varies greatly in the classification task. In the evaluation of pronunciation coherence in vocal music training, there may be a situation where the number of samples in some categories is much larger than that in other categories, which will cause the model to be biased towards the categories with a large number of samples during the training process, thus affecting the accuracy of the evaluation. Focal Loss balances the difference in the number of samples by setting different weights for samples in different categories. The formula of Focal Loss is:

[0063]

[0064] where \(\alpha\) is the balance factor used to adjust the weights of different categories; \(\gamma\) is the focusing parameter used to control the attention degree to easy-to-classify samples and difficult-to-classify samples; \(p\) i is the probability that the model predicts that the sample belongs to the \(i\)-th class. By adjusting the values of \(\alpha\) and \(\gamma\), the model can pay more attention to the categories with fewer samples, thereby improving the performance of the model in the case of class imbalance.

[0065] Embodiment 3:

[0066] In step 3, the nodes of the graph neural network include respiratory rate, peak value of laryngeal electromyogram signal, and pitch variance. The respiratory rate reflects the breathing rhythm of the singer during vocalization, which is closely related to the supply of breath; the peak value of laryngeal electromyogram signal can reflect the degree of muscle force in the larynx during vocalization; the pitch variance represents the fluctuation of pitch and is an important indicator to measure pitch stability.

[0067] The edge weights are calculated through the mutual information algorithm. Mutual information is an index used to measure the degree of mutual dependence between two random variables. In this embodiment, the mutual information between different nodes (such as respiratory rate and pitch variance) is calculated to determine the edge weights. The greater the mutual information, the stronger the correlation between the two nodes, and the greater the edge weights. The calculation formula of mutual information is:

[0068]

[0069] Among them, X and Y are two random variables, p(x, y) is their joint probability distribution, and p(x) and p(y) are their marginal probability distributions respectively.

[0070] The model updates the node states using a message passing mechanism. The message passing mechanism is one of the core mechanisms of graph neural networks. It updates the feature representations of nodes by passing messages between nodes. In each iteration, each node updates its own state based on the features and edge weights of its neighboring nodes. For example, node i receives messages from neighboring node j. The content of the messages includes the features and edge weights of node j, and then aggregates these messages according to certain rules to update its own feature representation.

[0071] The pronunciation efficiency comprehensive score is output through graph pooling operations. Graph pooling operations are similar to pooling operations in convolutional neural networks. They can aggregate the nodes of a graph, reduce the scale of the graph, and at the same time retain the important features of the graph. In this embodiment, the features of all nodes in the graph neural network are aggregated through graph pooling operations, and finally a comprehensive score is output. This score can reflect the pronunciation efficiency of the singer during the vocalization process. For example, methods such as average pooling or max pooling can be used to aggregate the node features, and then the aggregated features are mapped to a specific score value through a fully connected layer.

[0072] Embodiment 4:

[0073] This embodiment elaborates in detail the specific implementation method of the conditional adversarial generation network in constructing a pronunciation scenario simulator, and how to enhance the cross-style generalization ability of the model by generating synthetic speech data sets with different vocal styles and domain adaptation methods, so that the evaluation system can adapt to the training and evaluation of multiple vocal styles. The specific steps include:

[0074] In step 4, the generator of the conditional adversarial generation network adopts a U-Net structure. The U-Net structure is a neural network structure widely used in the field of image segmentation. It has an encoder and a decoder, and there are skip connections between the encoder and the decoder. In this embodiment, the generator with the U-Net structure can effectively learn the feature representations of voices with different vocal styles and generate high-quality synthetic speech data.

[0075] The discriminator introduces spectral normalization constraints. Spectral normalization is a technique for normalizing the discriminator. It can make the discriminator train more stably and improve the quality of the generated data. Through spectral normalization constraints, the discriminator can more accurately judge the differences between the generated data and the real data.

[0076] The conditional vector includes singing style tags and the singer's vocal range characteristics. Singing style tags (such as bel canto, pop, and folk singing) and the singer's vocal range characteristics (such as high pitch, middle pitch, and low pitch) are input into the conditional adversarial generation network as the conditional vector to guide the generator to generate synthetic speech data with specific styles and vocal ranges. In this way, the generator can generate diverse speech data according to different conditions to meet the needs of training and evaluation of different vocal styles.

[0077] The loss function adopts a combination of the Wasserstein distance and the gradient penalty term. The Wasserstein distance can better measure the difference between two probability distributions. Compared with the loss function of the traditional adversarial generation network, it can make the training more stable and the generated data of higher quality. The gradient penalty term is used to solve the problems of vanishing gradients or exploding gradients that may occur during the training process to ensure the training effect of the discriminator. The formula for the Wasserstein distance is:

[0078]

[0079] where P r is the distribution of real data, P g is the distribution of generated data, ∏(P r , P g ) is all possible joint distributions between P r and P g , γ is one of the joint distributions, (x, y) is a sample pair sampled from γ, and ‖x - y‖ is the distance between the sample pairs. The formula for the gradient penalty term is:

[0080]

[0081] where λ is the gradient penalty coefficient, is the data distribution obtained by linearly interpolating between real data and generated data, is the data sampled from , is the output of the discriminator for , is the gradient of the discriminator for . By minimizing this combined loss function, the generator and the discriminator can be continuously optimized to generate a synthetic speech data set closer to real speech data.

[0082] In practical applications, first, a large amount of real recording data of different singers' bel canto, pop, and ethnic singing styles is collected as training data. Then, this data is input into a conditional adversarial generation network for training. During the training process, the generator generates synthetic speech data based on the conditional vector, and the discriminator differentiates between the generated data and the real data. By continuously adjusting the parameters of the generator and the discriminator, the data generated by the generator becomes closer and closer to the real data, and at the same time, the discriminator can more accurately judge the authenticity of the data. After a certain number of trainings, the generator can generate a high-quality synthetic speech dataset containing different vocal styles. These synthetic speech datasets, together with the real recording data, align the feature distributions through domain adaptation methods, providing rich data support for subsequent model training and evaluation, and enhancing the cross-style generalization ability of the model.

[0083] Embodiment 5:

[0084] In step 5, the Shapley value decomposition algorithm approximates the marginal contribution through Monte Carlo sampling to perform local interpretability analysis on the non-linear evaluation model. Monte Carlo sampling is a method of estimating mathematical expectations through random sampling. In this embodiment, by randomly sampling different feature subsets multiple times, the contribution of each feature subset to the model prediction result is calculated, thereby approximating the Shapley value. The Shapley value can measure the importance of each feature in the model, and its calculation formula is:

[0085]

[0086] where φ i is the Shapley value of feature i, F is the set of all features, S is a subset of F, |S| represents the number of elements in subset S, and v(S) is the model prediction value corresponding to the feature subset S. By calculating the Shapley value of each feature, the influence weights of evaluation parameters such as pitch deviation, breath control, and resonance efficiency on the comprehensive score can be understood.

[0087] The hierarchical clustering algorithm is used to group and reduce the dimension of highly correlated parameters. The hierarchical clustering algorithm is a clustering method based on data similarity. It can merge similar data points into clusters and form a tree-shaped clustering structure. In this embodiment, by calculating the correlation between evaluation parameters, the parameters with higher correlation are grouped together. This can reduce the number of evaluation indicators, lower the complexity of the model, and retain important information at the same time. For example, methods such as Euclidean distance or Pearson correlation coefficient can be used to calculate the similarity between parameters, and then the hierarchical clustering algorithm is used to cluster the parameters. During the clustering process, the clustering threshold can be set according to actual needs to divide the parameters into different groups. Finally, one or several representative parameters are selected from each group as the core evaluation indicators to construct a lightweight evaluation indicator system. In this way, on the premise of ensuring the evaluation accuracy, the efficiency of the evaluation system can be improved, and the consumption of computing resources can be reduced.

[0088] In actual operation, it is assumed that a vocal training evaluation model including multiple evaluation parameters has been constructed. First, the Shapley value decomposition algorithm is used to calculate the Shapley value of each evaluation parameter through Monte Carlo sampling, and the influence weights of parameters such as pitch deviation, breath control, and resonance efficiency on the comprehensive score are obtained. Then, according to the correlation between these parameters, the hierarchical clustering algorithm is used to group them. For example, it is found that the correlation between certain parameters related to breath control is relatively high, and they are grouped together. Then, the most representative parameter is selected from each group as the core evaluation indicator. For example, in the group related to breath control, the breathing frequency is selected as the core evaluation indicator. In this way, a lightweight evaluation indicator system is constructed. This evaluation indicator system can be used to evaluate more efficiently in subsequent vocal training evaluations, while ensuring the accuracy and reliability of the evaluation results.

[0089] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device.

[0090] Although the embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for intelligent evaluation of vocal training based on speech analysis, characterized in that: The following steps are involved: Step 1: Obtain the original vocal training data, including audio signals, pitch sequences, rhythm features, and pronunciation physiological signals; when the amount of data is insufficient, adopt a small sample completion method based on transfer learning, use a pre-trained speech encoder to extract potential features, combine a variational autoencoder to generate and complete the missing rhythm features and timbre parameters, and align multimodal data through a dynamic time warping algorithm; Step 2: Based on the completed multimodal data, a pronunciation stability assessment model is constructed. With the fundamental frequency trajectory, formant distribution and glottal closure rate as the core features, a dynamic temporal convolutional network is used to capture the temporal dependency, and the pronunciation coherence is quantitatively modeled in combination with the attention mechanism. An adversarial training strategy is introduced to improve the model's robustness to noise. Step 3: Integrate physiological sensor data to establish a comprehensive pronunciation effectiveness evaluation model; through the synchronous collection of respiratory frequency, laryngeal muscle electrical signals and chest pressure data, use multi-source heterogeneous data fusion technology to build a dynamic relationship model based on graph neural network to quantify the association weight between respiratory support and pitch stability; Step 4: Construct a pronunciation scenario simulator for different vocal styles; use a conditional adversarial generative network to generate a synthetic speech dataset containing bel canto, pop, and folk singing, and align the feature distribution of real recordings and synthetic data through a domain adaptation method; Step 5: Conduct multi-dimensional sensitivity analysis on the evaluation parameters; The Shapley value decomposition algorithm is used to identify the influence weights of pitch deviation, breath control and resonance efficiency on the comprehensive score, and the core evaluation indicators are selected based on the feature importance ranking to build a lightweight evaluation indicator system. Step 6: Establish a vocal training program optimization agent model; Gaussian process regression is used to construct a pronunciation improvement path prediction model, and the Bayesian optimization algorithm is used to perform hierarchical optimization of training intensity, practice duration and phonation position parameters. Short-term modifiable parameters are optimized first, and then long-term training goals are iteratively adjusted.

2. The method for intelligently evaluating vocal training based on speech analysis according to claim 1, wherein: In the step 1, the small sample completion method of transfer learning specifically includes: using the pre-trained Wav2Vec 2.0 model to extract the context embedding vector of the speech signal, mapping the missing rhythm features to the latent space, generating the completion data through the VAE decoder, and using the contrastive learning loss function to constrain the distribution consistency of the generated data and the real data.

3. The method for intelligent assessment of vocal training based on speech analysis according to claim 1, characterized in that: In step 2, the dynamic time convolution network adopts an expanded convolution structure, the expansion coefficient increases exponentially, a gating mechanism is embedded between network layers, the output layer performs five-level classification of pronunciation coherence through a Softmax function, and Focal Loss is used to alleviate the problem of category imbalance.

4. The method for intelligently evaluating vocal training based on speech analysis according to claim 1, wherein: In step 3, the nodes of the graph neural network include respiratory frequency, laryngeal electromyography signal peak value and pitch variance, the edge weights are calculated by the mutual information algorithm, the model uses a message passing mechanism to update the node status, and outputs a comprehensive pronunciation efficiency score through a graph pooling operation.

5. The method for intelligently evaluating vocal training based on speech analysis according to claim 1, wherein: In step 4, the generator of the conditional adversarial generative network adopts a U-Net structure, the discriminator introduces a spectrum normalization constraint, the conditional vector includes a singing type label and a singer's vocal range feature, and the loss function adopts a combination of Wasserstein distance and a gradient penalty term.

6. The method for intelligently evaluating vocal training based on speech analysis according to claim 1, characterized in that: In step 5, the Shapley value decomposition algorithm approximates the marginal contribution through Monte Carlo sampling, performs local interpretability analysis on the nonlinear evaluation model, and uses a hierarchical clustering algorithm to group and reduce the dimension of highly correlated parameters.

7. The method for intelligently evaluating vocal training based on speech analysis according to claim 1, characterized in that: In step 6, the kernel function of Gaussian process regression adopts a combination of a Maltern 5 / 2 kernel and a periodic kernel, the hyperparameters are optimized by maximum likelihood estimation, the acquisition function of Bayesian optimization adopts an improved upper confidence bound strategy, and a dynamic balance coefficient is introduced to adjust the weight of exploration and utilization.

8. The method for intelligent assessment of vocal training based on speech analysis according to claim 1, characterized in that: It also includes a real-time feedback module, which specifically updates the evaluation model parameters through online incremental learning, uses a streaming computing framework to process real-time audio streams, and uses three-dimensional sound field visualization technology to display the distribution of pronunciation weaknesses in the feedback results.

9. The method for intelligently evaluating vocal training based on speech analysis according to claim 8, characterized in that: The three-dimensional sound field visualization technology uses spherical harmonics to decompose the propagation direction of sound waves, combines head-related transfer functions to generate a virtual auditory scene, and superimposes the weak point area on the vocal tract anatomical model through a heat map.

10. A vocal training intelligent evaluation system based on speech analysis, characterized in that: include: Data acquisition and completion module: used to obtain the original data of vocal training, including audio signals, pitch sequences, rhythm features and pronunciation physiological signals. When the amount of data is insufficient, a small sample completion method based on transfer learning is adopted. The pre-trained speech encoder is used to extract potential features, and the variational autoencoder is combined to generate and complete the missing rhythm features and timbre parameters, and the dynamic time warping algorithm is used to align the multimodal data. Pronunciation stability assessment module: Based on the completed multimodal data, with fundamental frequency trajectory, formant distribution and glottal closure rate as the core features, a dynamic temporal convolutional network is used to capture the temporal dependency, and the attention mechanism is combined to quantitatively model the pronunciation coherence. An adversarial training strategy is introduced to improve the model's robustness to noise, and a pronunciation stability assessment model is constructed; Comprehensive pronunciation effectiveness evaluation module: used to integrate physiological sensor data, synchronously collect respiratory frequency, laryngeal muscle electrical signals and chest pressure data, use multi-source heterogeneous data fusion technology to build a dynamic relationship model based on graph neural network, quantify the correlation weight between respiratory support and pitch stability, and establish a comprehensive pronunciation effectiveness evaluation model; Pronunciation scene simulation module: for different vocal styles, a conditional adversarial generative network is used to generate synthetic speech datasets including bel canto, pop and folk singing. The feature distribution of real recordings and synthetic data is aligned through domain adaptation methods to enhance the model's cross-style generalization ability and build a pronunciation scene simulator. Evaluation parameter analysis module: conducts multi-dimensional sensitivity analysis on evaluation parameters, uses Shapley value decomposition algorithm to identify the influence weight of pitch deviation, breath control and resonance efficiency on the comprehensive score, selects core evaluation indicators based on feature importance ranking, and constructs a lightweight evaluation indicator system; Training program optimization module: Gaussian process regression is used to construct a pronunciation improvement path prediction model, and the Bayesian optimization algorithm is combined to perform hierarchical optimization of training intensity, practice duration and vocalization position parameters, giving priority to optimizing short-term modifiable parameters, and then iteratively adjusting long-term training goals to establish a vocal training program optimization agent model.

Images