Objective evaluation method and system for voice timbre

Abstract

The present application relates to the technical field of speech quality evaluation, and particularly relates to a speech timbre objective evaluation method and system, comprising: collecting broadcasting professional pronunciation data and artistic speech data, obtaining a labeled timbre data set and an unlabeled artistic speech data set after preprocessing; constructing a TimbreMask model composed of an encoder and a decoder, and obtaining a trained TimbreMask timbre classification model by using the data set for segmented training; constructing a Fisher acoustic discriminant model, training the Fisher acoustic discriminant model with the labeled timbre data set, and obtaining a trained Fisher acoustic discriminant model; inputting the speech audio to be evaluated into the TimbreMask timbre classification model and the Fisher acoustic discriminant model respectively, and obtaining a comprehensive evaluation result. The present application constructs a speech timbre objective evaluation method which takes into account classification accuracy and physical interpretability.

Description

Technical Field

[0001] This invention relates to the field of speech quality evaluation technology, and in particular to a method and system for objective evaluation of speech timbre. Background Technology

[0002] Voice timbre is one of the core dimensions of sound perception. In artistic voice fields such as broadcasting, hosting, and recitation, timbre is usually described and evaluated by breaking it down into multiple perceptual dimensions such as thickness, brightness, and clarity. For a long time, voice timbre evaluation has mainly relied on subjective scoring by professional auditory experts on these dimensions. This method is heavily influenced by the expert's personal state and experience, making it difficult to standardize evaluation criteria and widely apply to broadcasting teaching practice. With the development of signal processing technology, objective evaluation methods based on traditional acoustic parameters such as fundamental frequency, formants, and harmonic noise ratio have been gradually researched and applied. However, these methods essentially perform linear analysis of acoustic signals, limiting their ability to characterize the timbre perception dimension, resulting in poor consistency between the evaluation results and human subjective perception.

[0003] In recent years, deep learning methods, represented by convolutional neural networks, recurrent neural networks, and self-supervised pre-trained models, have been gradually introduced into timbre classification and evaluation tasks. General-purpose self-supervised pre-trained speech models (such as Wav2vec 2.0 and AudioMAE) can learn rich speech representations through pre-training on large-scale speech data, achieving good results in conventional speech recognition and classification tasks. However, existing general-purpose pre-trained models are mainly designed for large-scale general speech data construction and lack adaptability to artistic speech scenarios with richer energy distributions and more complex spectral features. They struggle to learn sufficient and effective feature representations for professional timbre feature recognition tasks. Furthermore, deep learning models generally suffer from the "black box" problem; the model cannot identify which frequency band or physical characteristic causes the timbre to be classified into a certain category, making it difficult to directly apply these technologies to broadcasting teaching guidance.

[0004] Chinese patent document CN114067835A discloses a method for evaluating singing timbre similarity based on a two-dimensional singing timbre model. This method describes timbre as having two dimensions: "bright-dim" and "full-thin." It extracts classic audio features such as MFCC, spectral centroid, and LPC, as well as deep audio features extracted based on convolutional neural networks, to train a regression model that quantifies timbre in two-dimensional space. Timbre similarity is then calculated using Euclidean distance. While this method employs an end-to-end deep learning framework for feature extraction, the model itself struggles to explain its judgment criteria. It cannot explain to users which frequency band's energy distribution or acoustic characteristic led to the timbre being categorized. The evaluation results can only be presented in numerical or label form, making it difficult to translate into practical acoustic feedback, thus significantly limiting its application value in professional speech teaching scenarios. Summary of the Invention

[0005] In view of this, the present invention provides an objective evaluation method and system for speech timbre, in order to solve the problems in the prior art where speech timbre evaluation relies too much on subjective auditory expert judgment, the evaluation standards are not uniform and difficult to apply on a large scale, and the evaluation results of the existing deep learning methods lack interpretability, cannot explain the specific acoustic judgment basis to users, and are difficult to use for professional speech teaching guidance.

[0006] The technical solution of this invention is implemented as follows: On the one hand, the present invention provides a method for objectively evaluating speech timbre, including the following steps: S1. Collect professional broadcasting pronunciation data and label it with timbre tags. After preprocessing, obtain a labeled timbre dataset. Collect artistic speech data and preprocess it to obtain an unlabeled artistic speech dataset. S2. Construct a TimbreMask model consisting of an encoder and a decoder. Use an unlabeled art speech dataset to perform self-supervised pre-training on the TimbreMask model. Then, use a labeled timbre dataset to fine-tune the pre-trained TimbreMask model for multi-label timbre classification to obtain the trained TimbreMask timbre classification model. S3. Construct a Fisher acoustic discrimination model. Train the Fisher acoustic discrimination model based on the Fisher discrimination criterion using a labeled timbre dataset. Learn the contribution weight of each frequency band energy to timbre perception and obtain the trained Fisher acoustic discrimination model. S4. Input the audio to be evaluated into the TimbreMask timbre classification model and the Fisher acoustic discrimination model respectively, and fuse the output results of the two models to generate a comprehensive evaluation result that includes timbre level, rating score and vocal technique suggestions.

[0007] Based on the above technical solutions, preferably, the preprocessing in step S1 includes: resampling, noise reduction, pre-emphasis and normalization processing of the collected audio data in sequence, and uniformly converting the processed audio into mono format for storage; performing frame-by-frame windowing processing on the preprocessed audio, generating Log Mel spectrogram through Mel filter bank, and obtaining labeled timbre dataset and unlabeled artistic speech dataset for training.

[0008] Based on the above technical solutions, preferably, the TimbreMask model in step S2 includes an encoder and a decoder; the encoder is a feature extraction network based on the visual Transformer architecture; the decoder is a multi-layer Transformer structure, in which a sliding window-based local attention mechanism is used to group spectrogram patches into local windows for self-attention calculation, and the attention windows are offset between adjacent decoder layers to reconstruct the masked spectrogram patches.

[0009] Based on the above technical solutions, preferably, the training of the TimbreMask model in step S2 includes: S21. The Log Mel spectrogram of the unlabeled art speech dataset is divided into several spectrogram patches. The spectrogram patch sequence is masked using an unstructured random masking strategy. The feature vectors corresponding to the visible spectrogram patches are input into the encoder for feature extraction. The decoder then reconstructs the masked spectrogram patches and performs self-supervised pre-training with mean squared error as the loss function. S22. Based on the pre-trained encoder, a classification network is constructed. The Log Mel spectrogram of the labeled timbre dataset is used as input. The spectrogram is masked by a structured random masking strategy that combines the time axis and the frequency axis. The three timbre dimensions of thickness, brightness, and realism are simultaneously trained for multi-label classification using binary cross-entropy as the loss function, and the trained TimbreMask timbre classification model is obtained.

[0010] Based on the above technical solutions, preferably, the masking ratio of the unstructured random masking strategy in step S21 is 0.8; when the decoder reconstructs the masked spectrogram, it shifts the attention window by 50% between adjacent decoder layers and fills the boundary blanks that appear after the shift by cyclically shifting the spectrogram to the upper left.

[0011] Based on the above technical solutions, preferably, the structured random masking strategy in step S22 is to perform random masking with a masking ratio of 0.2 independently on the time axis and frequency axis of the spectrogram respectively; in the multi-label classification training, a weighted sampling method is used to deal with the class imbalance problem in the training samples, and a mixup data augmentation method is used to process the training samples.

[0012] Based on the above technical solutions, preferably, the construction and training of the Fisher acoustic discrimination model in step S3 includes: S31. The audio in the labeled timbre dataset is subjected to endpoint detection and amplitude normalization. The time-frequency representation is calculated by short-time Fourier transform, and then the frequency axis is mapped to the perceptual frequency space by Mel transform. The frequency range of 0~4280Hz is divided into 12 frequency bands according to the Mel equidistant principle. The energy of each frequency band is calculated and normalized to construct a frequency ratio feature vector. S32. Using the labeled timbre dataset, train the frequency proportion feature vector based on the Fisher discrimination criterion, with the optimization objective of minimizing intra-class divergence and maximizing inter-class divergence, and learn the contribution weight parameters of each frequency band to each timbre dimension of thickness, brightness, and solidity / virtuality; train the corresponding weight parameters for the two rating intervals of level 1-2 and level 2-3 under the same timbre dimension respectively; normalize the frequency band weight vector obtained after training to obtain the trained Fisher acoustic discrimination model.

[0013] Based on the above technical solutions, preferably, the energy of each frequency band mentioned in step S31 By examining the first The sum of the squares of the amplitudes of all time-frequency units within a frequency band is obtained, i.e.: ; in, For frequency band indexing, For the first The set of all time-frequency units within a frequency band. This is the complex time-frequency representation of the audio signal at time t and frequency f after the short-time Fourier transform; The frequency ratio feature vector Each component , It is the sum of the energy across all frequency bands.

[0014] Based on the above technical solutions, preferably, in the Fisher discrimination criterion described in step S32, the intra-class divergence... and inter-class divergence They are respectively: ; ; in, This represents the total number of timbre categories. For the first A sample set of classes This is a frequency-proportional feature vector for a single sample. For the first The feature mean vector of the class samples, For the first The number of samples in each class The mean of the population characteristic of all samples. The symbol for vector transpose; The normalization process for the frequency band weight vector is as follows: ; in, For the training obtained the first The original weights of each frequency band, This represents the total number of frequency bands. Continuous mapping values ​​for each timbre dimension The calculation is as follows: ; in, The first of the frequency proportional eigenvectors Each component.

[0015] In addition, the present invention also provides an objective evaluation system for speech timbre, the system being used to implement the aforementioned objective evaluation method for speech timbre, comprising: The data acquisition and preprocessing module is used to collect professional pronunciation data and artistic speech data, preprocess the collected audio data, and generate labeled timbre datasets and unlabeled artistic speech datasets. The TimbreMask timbre classification module is used to build the TimbreMask model. It is pre-trained and fine-tuned based on the unlabeled artistic speech dataset and the labeled timbre dataset, and performs timbre classification on the input speech audio to be evaluated, outputting the level results of each timbre dimension. The Fisher acoustic discrimination module is used to build the Fisher acoustic discrimination model. It is trained based on a labeled timbre dataset, performs frequency band energy analysis on the input speech audio to be evaluated, and outputs rating scores for each timbre dimension. The comprehensive evaluation output module is used to integrate the output results of the TimbreMask timbre classification module and the Fisher acoustic discrimination module to generate a comprehensive evaluation result that includes timbre level, rating score and vocal technique suggestions.

[0016] The present invention has the following advantages over the prior art: (1) This invention combines a self-supervised pre-trained classification model with a Fisher acoustic discrimination model based on frequency ratio features to construct an objective evaluation method for speech timbre that balances classification accuracy and physical interpretability. This method overcomes the dual limitations of inconsistent subjective evaluation standards and the lack of interpretability of deep learning model evaluation results in existing technologies. It can output a comprehensive evaluation result that includes timbre level, rating score, and vocal technique suggestions, and can be used for objective evaluation and teaching guidance of timbre in professional speech fields such as broadcasting, hosting, and recitation.

[0017] (2) In this invention, the TimbreMask model uses an art speech dataset instead of a general speech dataset for self-supervised pre-training, enabling the encoder to learn energy-rich and diverse feature representations in the art speech spectrum. Compared to pre-trained models trained on general speech datasets, it has stronger adaptability in downstream tasks of Chinese art speech timbre classification. In the fine-tuning stage, the decoder is removed, and only the classification network built by the encoder is fine-tuned. Even with small sample labeled data, it can still achieve high accuracy in classifying the three timbre dimensions of thickness, brightness, and realism.

[0018] (3) In this invention, the TimbreMask model's decoder employs a sliding window-based local attention mechanism. It groups spectrogram patches into local windows for self-attention calculation and shifts the attention windows between adjacent decoder layers. This fully utilizes the local correlation of spectrograms in the time-frequency domain, effectively improving the model's ability to reconstruct subtle timbre textures. Furthermore, the fine-tuning stage uses a structured random masking strategy combining the time and frequency axes. Compared to single-axis masking, this helps the model to more comprehensively understand the time-frequency features of artistic speech, further enhancing the generalization ability and robustness of the classification model.

[0019] (4) In this invention, the Fisher acoustic discrimination model uses the frequency proportion feature vector as input and directly establishes a linear mapping relationship between the relative proportion of energy in each Mel frequency band and the timbre perception dimension. The frequency band weight vector output by the model, after normalization, can intuitively present the contribution ratio of each frequency range to timbre perception, and has a clear physical meaning. Compared with end-to-end deep learning methods, the judgment criteria of this model are interpretable and can explain the energy distribution state of specific frequency bands to users, providing a quantifiable basis for acoustic feedback in broadcasting teaching.

[0020] (5) The present invention adopts a multi-label joint training method, and uses binary cross-entropy as the loss function to classify and train the three timbre dimensions of thickness, brightness and darkness, and virtual and real simultaneously. This allows the model to take into account the correlation between timbre dimensions during training, avoids the extra computational overhead caused by training three single-dimensional models independently, improves inference efficiency, and helps to alleviate the overfitting problem under small sample labeled data conditions. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1This is a flowchart of the objective evaluation method for voice timbre of the present invention; Figure 2 This is a structural diagram of the TimbreMask model of the present invention; Figure 3 This is a diagram showing the overall evaluation results of the present invention. Detailed Implementation

[0023] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0024] like Figure 1 As shown, the present invention provides a method for objective evaluation of speech timbre, comprising the following steps: S1. Collect professional broadcasting pronunciation data and label it with timbre tags. After preprocessing, obtain a labeled timbre dataset. Collect artistic speech data and preprocess it to obtain an unlabeled artistic speech dataset. S2. Construct a TimbreMask model consisting of an encoder and a decoder. Use an unlabeled art speech dataset to perform self-supervised pre-training on the TimbreMask model. Then, use a labeled timbre dataset to fine-tune the pre-trained TimbreMask model for multi-label timbre classification to obtain the trained TimbreMask timbre classification model. S3. Construct a Fisher acoustic discrimination model. Train the Fisher acoustic discrimination model based on the Fisher discrimination criterion using a labeled timbre dataset. Learn the contribution weight of each frequency band energy to timbre perception and obtain the trained Fisher acoustic discrimination model. S4. Input the audio to be evaluated into the TimbreMask timbre classification model and the Fisher acoustic discrimination model respectively, and fuse the output results of the two models to generate a comprehensive evaluation result that includes timbre level, rating score and vocal technique suggestions.

[0025] In one embodiment of the present invention, step S1 includes: sequentially resampling, noise reduction, pre-emphasis and normalization processing of the collected audio data, and uniformly converting the processed audio into mono format for storage; performing frame-by-frame windowing processing on the preprocessed audio, generating a Log Mel spectrogram through the Mel filter bank, and obtaining a labeled timbre dataset and an unlabeled artistic speech dataset for training.

[0026] Specifically, the dataset used in this invention consists of two parts. The first part is a labeled timbre dataset, composed of pronunciation data collected by broadcasting students in natural conditions, with the original collection standard being 44.1kHz, 16bit, and stereo. Auditory experts graded the collected data according to three timbre dimensions: thickness, brightness, and liveness, forming the labeled timbre dataset. The second part is an unlabeled artistic speech dataset, sourced from artistic speech fields such as news broadcasting, speeches, recitations, and hosting, totaling approximately 800 hours, and does not include timbre grade labels.

[0027] The two data sets mentioned above are preprocessed in a unified manner: resampling, noise reduction, pre-emphasis, and normalization are performed sequentially, and the processed audio is uniformly converted to 16kHz, 16bit, mono format for storage. Subsequently, the preprocessed audio is subjected to frame-by-frame windowing using a 25ms Hann window and a 10ms frame shift, and a Log Mel spectrogram is generated using a 128-dimensional Mel filter bank. The final spectrogram size is [size missing]. We obtained labeled timbre datasets and unlabeled artistic speech datasets for subsequent training.

[0028] In one embodiment of the present invention, the TimbreMask model in step S2 is based on an extension of the Masked Autoencoder (MAE) architecture, and consists of an encoder and a decoder, as follows: Figure 2 As shown, Figure 2 This is a schematic diagram of the overall structure of the TimbreMask model of this invention, showing the three main components: the Encode (ViT-B) encoder, the Decode (Transformer Block × 16) decoder, and the Classification Head. It also illustrates the masking strategies, loss functions (MSE and CE / BCE) used in the pre-training and fine-tuning stages, and the data flow. In the input representation and patch transformation stage, a default 10-second mono audio file is used as input to generate... The Log Mel spectrogram was divided into several spectrogram patches using non-overlapping convolutions with kernel size (16, 16) and stride (16, 16). The 10-second sample was obtained after this segmentation. Each patch is mapped to a Transformer token and then a fixed sine position code is added.

[0029] The encoder uses a 12-layer ViT-Base (ViT-B) backbone network, which is a feature extraction network based on the visual Transformer architecture. The encoder only receives the feature vectors corresponding to visible patches as input, and discards a large number of masked patches through a high masking ratio, thereby reducing the computational complexity of the Transformer and forcing the model to learn more discriminative semantic representations.

[0030] The decoder consists of multiple layers of standard Transformer modules, configured with the following parameters: number of attention heads. Embedded Dimensions Feedforward network dimension The decoder has a depth of 16 layers. The decoder's task is to complete the encoder's output (covering only visible tokens) into a full sequence and reconstruct the original spectral patch values. The encoded output from the encoder, after restoring the original time-frequency order, is augmented with fixed-sine-position embeddings before being input into the decoder. A linear prediction head is located at the top of the decoder to reconstruct the input spectrogram.

[0031] Because spectrograms have strong local correlations—the absolute position of spectrogram features (such as harmonics, formants, and fricatives) directly determines the physical properties and semantic information of the sound—a local attention mechanism based on sliding windows is introduced into the decoder. The spectrogram blocks are grouped into local windows for self-attention calculation. Referring to the sliding window mechanism of the Swing Transformer, the attention window is shifted by 50% between adjacent decoder layers, and the boundary blanks that appear after the shift are filled by cyclically shifting the spectrogram to the upper left. Ablation experiments have verified the effectiveness of this mechanism.

[0032] Step S2, the training of the TimbreMask model, is divided into two stages: pre-training and fine-tuning, specifically including: Step S21, Pre-training Stage: Using the Log Mel spectrogram of the unlabeled artistic speech dataset as input, an unstructured random masking strategy is employed to mask the spectrogram patch sequence, with an optimal masking ratio of 0.8. Specifically, the unstructured random masking strategy does not distinguish between the frequency axis and the time axis, directly selecting several patches from the entire spectrogram patch sequence for masking according to the masking ratio. The masked patches exhibit no structural regularity in the time-frequency two-dimensional space, and the probability of each patch being selected is independent and equal. This strategy is suitable for the pre-training stage, enabling the model to learn the correlation between the global statistical characteristics and local details of the spectrogram under unconstrained random missing conditions, thereby forming a robust representation ability of the artistic speech spectrum. The feature vectors corresponding to the visible spectrogram patches are input into the encoder for feature extraction, and then the decoder reconstructs the masked spectrogram patches, performing self-supervised pre-training with mean squared error (MSE) as the loss function. The training process randomly traversed all audio files for a total of 33 epochs, with a batch size of 512 and a learning rate of 0.0002. The total training time on an RTX 4090D GPU was approximately 35 hours. This pre-training strategy is based on the principle of "small but excellent," using energy-rich and diverse artistic speech data, primarily in Chinese, as pre-training data. This allows the model to learn more spectral feature representations related to artistic speech. Compared to models such as AudioMAE and SpecMask trained using general speech datasets, this model performs better in downstream tasks of Chinese artistic speech recognition.

[0033] Step S22, Fine-tuning Stage: Remove the decoder and construct a classification network based on the pre-trained encoder, using the Log Mel spectrogram of the labeled timbre dataset as input. The masking strategy is switched to a structured random masking strategy combining the time and frequency axes. Specifically, random masking with a masking ratio of 0.2 is performed independently on both the time and frequency axes of the spectrogram. Ablation experiments demonstrate that this structured masking method allows the model to more fully understand the time-frequency features of artistic speech compared to single-axis masking. Specifically, the structured random masking strategy uses either the time or frequency axis as the basic unit for masking. For example, in time-axis masking, several time frames are randomly selected according to the masking ratio, and all spectrogram patches in the corresponding column of each time frame are masked. Similarly, in frequency-axis masking, several frequency intervals are randomly selected according to the masking ratio, and all spectrogram patches in the corresponding row of each frequency interval are masked. Compared to unstructured random masking, structured masking creates a large-scale continuous occlusion of the time or frequency dimensions in a single training iteration, forcing the model to predict under conditions where entire time or frequency information is missing. This helps the model to more fully understand the overall patterns of the time-frequency structure of artistic speech during the fine-tuning phase, thereby improving the generalization ability and robustness of the classification model. Using binary cross-entropy as the loss function, multi-label classification training is performed simultaneously on the three timbre dimensions of thickness, brightness, and realism. This allows the model to consider the correlation between different timbre dimensions while avoiding the computational overhead of training three independent models. To address the class imbalance problem in the training samples, a weighted sampling method is used; simultaneously, mixup data augmentation is employed to process the training samples, improving the model's generalization ability and robustness. The final trained TimbreMask timbre classification model achieves a classification accuracy of 99.6% across the three timbre dimensions of thickness, brightness, and realism, significantly outperforming mainstream self-supervised pre-trained speech models such as Wav2vec 2.0, AST, and the original AudioMAE.

[0034] The TimbreMask model mentioned above fully utilizes the temporal-frequency local correlation of spectrograms by introducing a local attention mechanism, thereby improving the ability to reconstruct subtle timbre textures. At the same time, it uses an art speech dataset instead of a general speech dataset for pre-training, enabling the model to have stronger feature learning in art speech scenarios with richer energy distributions. It can still achieve high accuracy in multi-dimensional timbre classification under small sample labeling conditions, and improves inference efficiency through multi-label joint training.

[0035] Speech timbre, as a highly complex perceptual attribute, essentially originates from the spectral modulation structure of the vocal tract system in response to airflow excitation. Different timbre perception dimensions (thickness, brightness, and density) manifest as differences in energy distribution across different frequency regions in the frequency domain. Compared to traditional parametric features such as MFCC, frequency proportion features directly reflect the relative proportion of energy in each perceptually relevant frequency band, possessing higher physical interpretability and perceptual consistency. On the other hand, timbre level labels exhibit clear continuity and hierarchical attributes. The Fisher discrimination criterion, by simultaneously minimizing intra-class variance and maximizing inter-class mean difference, provides a naturally suitable mathematical tool for constructing a continuous timbre discrimination space.

[0036] Based on this, in one embodiment of the present invention, step S3 combines frequency ratio features with Fisher's discrimination criterion to construct a timbre discrimination model with physical meaning and interpretability, specifically including: Step S31: Perform endpoint detection and amplitude normalization on the audio data in the labeled timbre dataset, and calculate the time-frequency representation using Short Time Fourier Transform (STFT). The STFT extraction parameters are as follows: , , The window function is a Hann window. To conform to the characteristics of human hearing perception, the frequency axis is mapped to the perceptual frequency space through Mel transform. Within the Hz range, it is divided according to the Mel equidistant principle into Each frequency band.

[0037] Within the 0–4280Hz frequency range, this interval encompasses the main formants, harmonic energy, and frictional noise components that contribute to timbre formation. Dividing this range into 12 Mel bands ensures that each band has approximately equivalent information capacity in auditory perception, stably reflecting trends in timbre variation. Too few bands will lead to excessive compression of the spectral structure, masking key differences; too many bands introduce redundant dimensions, making the model prone to overfitting and difficult to interpret. This represents an optimal trade-off between expressive power and model stability, and also aligns with the natural segmentation of frequency perception in human ear acoustics. Furthermore, considering the differences in frequency emphasis across different timbre dimensions, based on the initial 12 frequency bands, frequency bands exhibiting linear variations in energy proportion and level were manually selected according to the statistical performance of different dimensions and levels. The experimental results were then compared with those after adding additional feature dimensions such as total energy, spectral centroid, and formants. Ultimately, a suitable combination of feature dimensions was determined for each timbre dimension; therefore, the actual weighting dimensions used for each timbre dimension are not all kept at 12.

[0038] Energy in each frequency band By examining the first The summation of the squares of the amplitudes of all time-frequency units within a frequency band yields: ; in, For frequency band indexing, For the first The set of all time-frequency units within a frequency band. The audio signal after short-time Fourier transform at time 10:00 ,frequency Complex time-frequency representation at a given point. Construct a frequency-proportional eigenvector. Each component , It is the sum of the energies of all frequency bands. This represents the total number of frequency bands.

[0039] Step S32: Train the frequency proportion feature vector using the labeled timbre dataset based on the Fisher discriminant criterion. The optimization objective is to minimize the intra-class divergence and maximize the inter-class divergence. In the Fisher discriminant criterion, intra-class divergence... Inter-class divergence They are defined as follows: ; ; in, This represents the total number of timbre categories. For the first A sample set of classes This is a frequency-proportional feature vector for a single sample. For the first The feature mean vector of the class samples, For the first The number of samples in each class The population characteristic mean of all samples This is the vector transpose symbol. The training process aims to minimize the Fisher loss function. To optimize the objective, the model simultaneously reduced intra-class variance and increased inter-class mean. After training, the intra-class variance of the model generally converged to below 0.01, and the difference in inter-class means was greater than 0.25. The output score was positively correlated with the subjective timbre perception value, which can be directly interpreted as a continuous perception index.

[0040] Considering the differences in frequency energy distribution between the rating standards of levels 1-2 and 2-3 within the same timbre dimension, corresponding weight parameters are trained for the two rating intervals of levels 1-2 and 2-3 within the same timbre dimension. In actual inference, the corresponding weight parameters are selected for calculation based on the level judgment given by the TimbreMask timbre classification model.

[0041] After training, the obtained frequency band weight vector is normalized: ; in, For the training obtained the first The original weights of each frequency band, This represents the total number of frequency bands. The normalized weight vector. This directly characterizes the contribution ratio of each frequency band to the corresponding timbre perception dimension. Continuous mapping values ​​for each timbre dimension. The calculation is as follows: ; in, The first eigenvector of the frequency proportion One portion, It can intuitively explain the contribution mechanism of different frequency ranges to timbre perception.

[0042] The aforementioned Fisher acoustic discrimination model, through the parameterization of frequency proportion features, directly establishes a mapping relationship between the timbre perception dimension and the interpretable frequency band energy distribution. This preserves auditory interpretability while achieving efficient machine learning optimization, thus bridging the gap between acoustic analysis and psychoacoustic modeling. This method can be viewed as a "perception-driven discrimination mapping," realizing the transformation from a black-box model to an interpretable acoustic mechanism model, and providing quantifiable evidence for acoustic feedback in broadcasting instruction.

[0043] In one embodiment of the present invention, step S4 includes: After preprocessing the audio samples to be evaluated to generate Log Mel spectrograms, they are input into the TimbreMask timbre classification model and the Fisher acoustic discriminant model for inference. The TimbreMask timbre classification model outputs the level judgment of the audio sample in three timbre dimensions: thickness, brightness, and realism. The Fisher acoustic discriminant model selects corresponding weight parameters based on the level range output by the TimbreMask model and calculates the continuous mapping value of each timbre dimension. and will Mapped to a rating score of 1 to 100.

[0044] The grade assessment for each timbre dimension corresponds to the following standard rating: Level 1 corresponds to Grade C, Level 2 to Grade B, and Level 3 to Grade A. (Continuous mapping values) The actual distribution across each timbre dimension varies due to the statistical characteristics of the training data. This can be determined by analyzing samples at each level in the training set. Statistical analysis of value distribution was conducted to determine the range of each level. The maximum and minimum values ​​of the T-value are used to map the T-value to a rating score range of 1–100 using linear interpolation, i.e.: ; in, and Each is within the current level range in the training set. Minimum and maximum values, The mapped rating score ranges from 1 to 100.

[0045] The output format of the comprehensive evaluation results is a combination of standard ratings (A~C) and rating scores (1~100). C-level rating scores can be compared horizontally within the same level, while A and B-level rating scores can be compared horizontally with each other. The evaluation results also correlate three timbre dimensions—thickness / thinness, brightness / darkness, and liveness / virtualness—with specific vocal technique suggestions. Visualization is expected through statistical charts showing the energy proportions of different frequencies, using different color depths to represent the weight of each frequency range. An energy distribution chart of excellent voices will be provided for comparison, intuitively guiding users to understand the energy status of their own voice in each frequency band and identify areas for improvement. The visualization effect of the comprehensive evaluation results is as follows: Figure 3 As shown, the left side is a radar chart with resonance control, elasticity control, and breath control as the coordinate axes. Resonance control, elasticity control, and breath control correspond to the teaching application layer display formats of the thickness, brightness, and timbre dimensions in this invention, respectively. The shaded area reflects the evaluation results of the voice to be evaluated in each dimension. The right side is a list of voice training suggestions automatically generated based on the diagnostic results. Completed items are marked with checkmarks for user reference. Furthermore, this comprehensive evaluation result is provided to external applications via an API interface.

[0046] The aforementioned dual-model fusion evaluation strategy creatively combines the high accuracy of deep learning with the interpretability of Fisher's linear discrimination. It uses deep learning models to solve the problem of "hearing accurately" and Fisher models to calculate frequency band energy weights to solve the problem of "speaking clearly," filling the current technological gap where speech timbre evaluation cannot be used for teaching feedback.

[0047] This invention also provides an objective speech timbre evaluation system, which is used to implement the objective speech timbre evaluation method described above, including: The data acquisition and preprocessing module is used to collect professional pronunciation data and artistic speech data, preprocess the collected audio data, and generate labeled timbre datasets and unlabeled artistic speech datasets. The TimbreMask timbre classification module is used to build the TimbreMask model. It is pre-trained and fine-tuned based on the unlabeled artistic speech dataset and the labeled timbre dataset, and performs timbre classification on the input speech audio to be evaluated, outputting the level results of each timbre dimension. The Fisher acoustic discrimination module is used to build the Fisher acoustic discrimination model. It is trained based on a labeled timbre dataset, performs frequency band energy analysis on the input speech audio to be evaluated, and outputs rating scores for each timbre dimension. The comprehensive evaluation output module is used to integrate the output results of the TimbreMask timbre classification module and the Fisher acoustic discrimination module to generate a comprehensive evaluation result that includes timbre level, rating score and vocal technique suggestions.

[0048] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An objective evaluation method and system for voice timbre, comprising the following steps: S1. Collect professional broadcasting pronunciation data and label it with timbre tags. After preprocessing, obtain a labeled timbre dataset. Collect artistic speech data and preprocess it to obtain an unlabeled artistic speech dataset. S2. Construct a TimbreMask model consisting of an encoder and a decoder. Use an unlabeled art speech dataset to perform self-supervised pre-training on the TimbreMask model. Then, use a labeled timbre dataset to fine-tune the pre-trained TimbreMask model for multi-label timbre classification to obtain the trained TimbreMask timbre classification model. S3. Construct a Fisher acoustic discrimination model. Train the Fisher acoustic discrimination model based on the Fisher discrimination criterion using a labeled timbre dataset. Learn the contribution weight of each frequency band energy to timbre perception and obtain the trained Fisher acoustic discrimination model. S4. Input the audio to be evaluated into the TimbreMask timbre classification model and the Fisher acoustic discrimination model respectively, and fuse the output results of the two models to generate a comprehensive evaluation result that includes timbre level, rating score and vocal technique suggestions.

2. The objective evaluation method for voice timbre as described in claim 1, characterized in that, The preprocessing described in step S1 includes: resampling, noise reduction, pre-emphasis and normalization of the acquired audio data in sequence, and converting the processed audio into a mono format for storage; performing frame-by-frame windowing on the preprocessed audio, generating a LogMel spectrogram through the Mel filter bank, and obtaining a labeled timbre dataset and an unlabeled artistic speech dataset for training.

3. The objective evaluation method for voice timbre as described in claim 1, characterized in that, The TimbreMask model in step S2 includes an encoder and a decoder; the encoder is a feature extraction network based on the visual Transformer architecture; the decoder is a multi-layer Transformer structure, which uses a sliding window-based local attention mechanism to group spectrogram patches into local windows for self-attention calculation, and shifts the attention windows between adjacent decoder layers to reconstruct the masked spectrogram patches.

4. The objective evaluation method for voice timbre as described in claim 1, characterized in that, The training of the TimbreMask model in step S2 includes: S21. The Log Mel spectrogram of the unlabeled art speech dataset is divided into several spectrogram patches. The spectrogram patch sequence is masked using an unstructured random masking strategy. The feature vectors corresponding to the visible spectrogram patches are input into the encoder for feature extraction. The decoder then reconstructs the masked spectrogram patches and performs self-supervised pre-training with mean squared error as the loss function. S22. Based on the pre-trained encoder, a classification network is constructed. The Log Mel spectrogram of the labeled timbre dataset is used as input. The spectrogram is masked by a structured random masking strategy that combines the time axis and the frequency axis. The three timbre dimensions of thickness, brightness, and realism are simultaneously trained for multi-label classification using binary cross-entropy as the loss function, and the trained TimbreMask timbre classification model is obtained.

5. The objective evaluation method for voice timbre as described in claim 4, characterized in that, The masking ratio of the unstructured random masking strategy described in step S21 is 0.8; when the decoder reconstructs the masked spectrogram, it shifts the attention window by 50% between adjacent decoder layers and fills the boundary blanks that appear after the shift by cyclically shifting the spectrogram to the upper left.

6. The objective evaluation method for voice timbre as described in claim 4, characterized in that, The structured random masking strategy in step S22 involves performing random masking with a masking ratio of 0.2 independently on both the time axis and frequency axis of the spectrogram. In the multi-label classification training, a weighted sampling method is used to handle the class imbalance problem in the training samples, and a mixup data augmentation method is used to process the training samples.

7. The objective evaluation method for voice timbre as described in claim 1, characterized in that, Step S3, the construction and training of the Fisher acoustic discrimination model, includes: S31. The audio in the labeled timbre dataset is subjected to endpoint detection and amplitude normalization. The time-frequency representation is calculated by short-time Fourier transform, and then the frequency axis is mapped to the perceptual frequency space by Mel transform. The frequency range of 0~4280Hz is divided into 12 frequency bands according to the Mel equidistant principle. The energy of each frequency band is calculated and normalized to construct a frequency ratio feature vector. S32. Using the labeled timbre dataset, train the frequency proportion feature vector based on the Fisher discrimination criterion, with the optimization objective of minimizing intra-class divergence and maximizing inter-class divergence, and learn the contribution weight parameters of each frequency band to each timbre dimension of thickness, brightness, and solidity / virtuality; train the corresponding weight parameters for the two rating intervals of level 1-2 and level 2-3 under the same timbre dimension respectively; normalize the frequency band weight vector obtained after training to obtain the trained Fisher acoustic discrimination model.

8. The objective evaluation method for voice timbre as described in claim 7, characterized in that, The energy of each frequency band mentioned in step S31 By examining the first The sum of the squares of the amplitudes of all time-frequency units within a frequency band is obtained, i.e.: ； in, For frequency band indexing, For the first The set of all time-frequency units within a frequency band. The audio signal after short-time Fourier transform at time 10:00 t ,frequency f Complex time-frequency representation at a given point; The frequency ratio feature vector Each component , It is the sum of the energy across all frequency bands.

9. The method for objectively evaluating speech timbre as described in claim 7, characterized in that, In the Fisher criterion described in step S32, the intra-class divergence... and inter-class divergence They are respectively: ； ； ； in, This represents the total number of timbre categories. For the first A sample set of classes This is a frequency-proportional feature vector for a single sample. For the first The feature mean vector of the class samples, For the first The number of samples in each class The population characteristic mean of all samples The symbol for vector transpose. For the frequency band weight vector used in training; The normalization process for the frequency band weight vector is as follows: ; in, For the training obtained the first The original weights of each frequency band, This represents the total number of frequency bands. Continuous mapping values ​​for each timbre dimension The calculation is as follows: ; in, The first of the frequency proportional eigenvectors Each component.

10. An objective evaluation system for voice timbre, characterized in that, The system is used to implement the objective evaluation method for speech timbre as described in any one of claims 1-9, including: The data acquisition and preprocessing module is used to collect professional pronunciation data and artistic speech data, preprocess the collected audio data, and generate labeled timbre datasets and unlabeled artistic speech datasets. The TimbreMask timbre classification module is used to build the TimbreMask model. It is pre-trained and fine-tuned based on the unlabeled artistic speech dataset and the labeled timbre dataset, and performs timbre classification on the input speech audio to be evaluated, outputting the level results of each timbre dimension. The Fisher acoustic discrimination module is used to build the Fisher acoustic discrimination model. It is trained based on a labeled timbre dataset, performs frequency band energy analysis on the input speech audio to be evaluated, and outputs rating scores for each timbre dimension. The comprehensive evaluation output module is used to integrate the output results of the TimbreMask timbre classification module and the Fisher acoustic discrimination module to generate a comprehensive evaluation result that includes timbre level, rating score and vocal technique suggestions.

Images