A method and system for automatic speech quality assessment without reference speech
Through the reference-free speech quality evaluation method, deep learning model and multi-path sequence modeling technology are used to solve the accuracy and cost of the existing speech quality evaluation method, real-time quality monitoring and efficient evaluation in each stage of speech processing are achieved.
Patent Information
- Application Number
- CN202211144904.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-20
- Publication Date
- 2025-08-08
- Estimated Expiration
- 2042-09-20
AI Technical Summary
The existing speech quality evaluation methods require reference to voice, with low evaluation accuracy and high labor and time costs, so it is impossible to achieve real-time monitoring of the quality of each stage of speech processing.
By obtaining the speech data to be detected, extracting spectral features and acoustic coding features, using multi-path sequence modeling networks and deep learning models to model speech acoustic features, combining attention pooling and anti-normalization methods, predicting and mapping to the value space of the average opinion score to achieve reference-free speech quality evaluation.
It reduces the labor and time cost of listening and evaluating voice signals, improves the accuracy of prediction results, and can monitor the quality of each stage of voice processing in real time, and is suitable for multiple fields of voice signal processing.
Smart Images
Figure CN115457980B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of audio and intelligent computing technology, and in particular to an automatic speech quality assessment method and system without reference speech. Background Art
[0002] With the advancement of digital communication and deep learning technologies, speech processing has become increasingly complex, making speech quality evaluation increasingly important. Traditional speech quality assessment methods are divided into objective evaluation and subjective scoring. Objective evaluation methods, such as Perceptual Evaluation of Speech Quality (PESQ) and Perceptual Objective Listening Quality Analysis (POLQA), are based on reference speech. They primarily focus on the waveform and spectral differences between the reference and processed speech. Existing technologies cannot accurately reflect the quality of speech as perceived by the human ear. Therefore, subjective scoring is required, such as the Mean Opinion Score (MOS), which is the most commonly used evaluation method for speech quality. This method does not require a reference speech, but it requires significant labor and time, and cannot measure speech quality in a timely manner. For speech processing technologies such as speech synthesis, speaker extraction, and speech enhancement, the Mean Opinion Score is the most effective and necessary evaluation method for assessing the actual performance of the model. However, the original average subjective opinion scoring method requires manual scoring of each speech, resulting in serious lag in the method. Therefore, it cannot be used as a means of monitoring speech quality in real time at each stage of speech processing throughout the entire speech processing process.
[0003] Existing automated speech quality assessment methods have the following problems: they require reference speech, resulting in low assessment accuracy. Existing neural network-based automated speech quality assessment methods do not specify the range of output results. The design standard for the average subjective opinion score ranges from 1 to 5. Existing models directly convert values greater than 5 and less than 1 into 5 and 1, respectively. This fails to reflect the differences between speech sounds at the same cutoff score of 1 or 5 in the actual results, thereby reducing the accuracy of the prediction results. Summary of the Invention
[0004] The present invention aims to provide a method and system for automatic speech quality assessment without reference speech, so as to solve the technical problems in the prior art that existing speech quality assessment methods require reference speech and the prediction results are inaccurate through non-manual means.
[0005] The present invention solves the above problems through the following technical solutions:
[0006] An automated speech quality assessment method without reference speech comprises the following steps:
[0007] Step S1. Obtaining the voice data to be detected;
[0008] Step S2. extracting the spectral features of the speech to be detected;
[0009] Step S3. Encode the spectral features of the speech to be detected and extract the corresponding acoustic coding features;
[0010] Step S4: feeding the acoustic coding features into a multi-path sequence modeling network, modeling the temporal dependency of the acoustic coding features, and extracting speech acoustic features;
[0011] Step S5. Use the extracted speech acoustic features to predict the mean opinion score of the speech to be detected, and map the predicted normalized result to the value space of the mean opinion score through the denormalization method to obtain the final quality assessment result of the speech to be detected.
[0012] As a further improvement of the present invention, in step S2, Mel-spectrogram features are extracted as spectral features of the speech to be detected.
[0013] As a further improvement of the present invention, in step S3, the spectral features of the speech to be detected are encoded using an acoustic encoder module, and the acoustic encoder module is composed of a one-dimensional convolutional network and a residual network.
[0014] As a further improvement of the present invention, the one-dimensional convolutional network of the acoustic encoder module has an input dimension of 32 and an output dimension of 128;
[0015] The residual network consists of three residual network blocks. The parameters of the three residual network blocks are: the input dimension of the first residual network block is 128, and the output dimension is 128; the input dimension of the second residual network block is 128, and the output dimension is 256; the input dimension of the third residual network block is 256, and the output dimension is 256; each residual network block is composed of a one-dimensional convolution layer, a batch normalization network layer, a PReLU activation layer, a one-dimensional convolution layer, and a batch normalization network layer to form the first data path. The 1×1 convolution network is used as the residual path. The results of the first data path and the residual path are added, and the encoding result is obtained through the PReLU activation layer.
[0016] As a further improvement of the present invention, the multi-path sequence modeling network is a dual-path recurrent neural network, a gating-based dual-path recurrent neural network or a dual-path Transformer.
[0017] As a further improvement of the present invention, the multi-path sequence modeling network is arranged in a sequence modeling model, which includes layer normalization, a 1×1 convolutional network, a segmentation layer, and a 4-layer stacked dual-path recurrent neural network in sequence, and then connected to a PReLU activation layer, a two-dimensional convolutional network layer, and an overlap-addition layer, and then divided into two paths of data. The two paths of data are a 1×1 convolutional network and a Tanh activation function as a gating signal, and a 1×1 convolutional network and a Sigmoid as a signal to be processed. The gating operation of the signal to be processed is completed by multiplication, and then the speech acoustic features are obtained after being processed by a 1×1 convolutional network.
[0018] As a further improvement of the present invention, each layer of the dual-path recurrent neural network includes an intra-block processing module and an inter-block processing module.
[0019] The intra-block processing module consists of a pair of intra-block recurrent neural networks, which are quickly implemented using LSTM or GRU. The intra-block recurrent neural network processes along the second dimension of the input three-dimensional tensor. The output of the intra-block recurrent neural network is then point-by-point multiplied and concatenated with the input three-dimensional tensor of the same size. A linear layer is then used to map the output to a tensor of the same size as the input, which is then transposed to obtain the output three-dimensional tensor of the intra-block processing.
[0020] The inter-block processing module is structurally the same as the intra-block processing module. The inter-block recurrent neural network and splicing operations of the inter-block processing module are both processed along the third dimension.
[0021] As a further improvement of the present invention, step S5 is implemented by a speech quality assessment model, which includes an attention pooling network model and a denormalization model. The specific method is:
[0022] The extracted speech acoustic features pass through the network part of the attention pooling network model and then perform self-attention calculation. The attention weight is multiplied by the speech acoustic features and then passes through the fully connected layer. The output obtained after normalization of the attention pooling network model is mapped to the scoring range of the average opinion score. The final quality prediction evaluation result of the speech to be tested is obtained through the denormalization model.
[0023] As a further improvement of the present invention, the denormalization module is implemented according to the following formula:
[0024]
[0025] It includes the tanh activation layer and the denormalization method. The tanh calculation obtains a score between 0 and 1, and the denormalization method maps the normalized result to the value range of the subjective average opinion score. Here For the final scoring results, This is the scoring output of the attention pooling layer.
[0026] The present invention also solves the above-mentioned problem through the following technical solutions:
[0027] An automated speech quality assessment system for speech without reference, for implementing the automated speech quality assessment method for speech without reference as described above, comprising a speech collection module, an acoustic feature extraction module, and a speech quality scoring module, wherein the acoustic feature extraction module comprises a spectral feature extraction module, an acoustic encoder module, and a sequence modeling module;
[0028] A voice collection module is used to collect the acquired voice data to be detected in a storage device through an acoustic sensor;
[0029] The acoustic feature extraction module reads the speech data to be detected from the storage device, extracts the spectral features of the speech to be detected through the spectral feature extraction module, and then the acoustic encoder module encodes the spectral features of the speech to be detected based on the spectral features and extracts the corresponding acoustic coding features. The sequence modeling module models the time dependency of the acoustic coding features based on the acoustic coding features to obtain the acoustic features of the speech.
[0030] The speech quality scoring module includes an attention pooling network module and a denormalization module. The attention pooling network module predicts the mean opinion score of the speech acoustic features of the input speech to be detected, and maps the predicted normalized prediction results to the value space of the mean opinion score through the denormalization module to obtain the final quality assessment result of the speech to be detected.
[0031] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0032] The present invention sequentially extracts the spectral features and acoustic coding features of the speech data to be detected, and simultaneously feeds the acoustic coding features into a multi-path sequence modeling network for multi-path sequence modeling. It then uses a deep learning model to simulate the human ear's auditory evaluation of audio speech quality, and predicts the mean opinion score of the speech to be detected. Compared with traditional speech auditory evaluation, which relies solely on subjective human scoring, this method can significantly reduce the labor and time costs of speech signal auditory evaluation and improve the accuracy of the prediction results.
[0033] The present invention also provides an automated evaluation method, which can play a significant role in multiple speech signal processing fields, such as signal distortion evaluation of speech signal transmission in the communication field, evaluation of synthesized speech quality of speech synthesis models, and speech quality evaluation of speech processing results by speech extraction or speech separation models and speech enhancement models, so as to improve the accuracy of prediction results. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 Flowchart of the method for automatic speech quality assessment without reference speech of the present invention;
[0035] Figure 2 This is a flow chart of training an automated speech quality assessment model without reference speech according to the present invention;
[0036] Figure 3 This is a flow chart of the operation of the automatic speech quality assessment model without reference speech of the present invention;
[0037] Figure 4 Schematic diagram of the speech quality assessment model of the present invention;
[0038] Figure 5 This is a schematic diagram of a multi-layer stacked multi-path recurrent neural network model of the present invention;
[0039] Figure 6 Schematic diagram of the automatic speech quality assessment system without reference speech of the present invention. DETAILED DESCRIPTION
[0040] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0041] Example 1:
[0042] Combined with attachment Figure 1-5 As shown, an automatic speech quality assessment method without reference speech specifically includes the following steps:
[0043] Step S1. Obtaining the voice data to be detected;
[0044] Step S2. extracting the spectral features of the speech to be detected;
[0045] In this embodiment, the spectral features of the speech to be detected are extracted, and the spectral features of the speech to be detected are adopted but not limited to the extraction of Mel spectrum features. This embodiment is described by extracting Mel spectrum features from speech data as the spectral features.
[0046] Step S3. Encode the spectral features of the speech to be detected and extract the corresponding acoustic coding features;
[0047] In this embodiment, the acoustic encoder module is used to encode the spectral features of the speech to be detected to extract the acoustic coding features of the encoded spectral features;
[0048] The acoustic encoder module consists of a one-dimensional convolutional network and a residual network.
[0049] Among them, the input dimension of the one-dimensional convolutional network is 32 and the output dimension is 128;
[0050] The residual network consists of three residual network blocks. The parameters of the three residual network blocks are: the input dimension of the first residual network block is 128, and the output dimension is 128; the input dimension of the second residual network block is 128, and the output dimension is 256; the input dimension of the third residual network block is 256, and the output dimension is 256; each residual network block is composed of a one-dimensional convolution layer, a batch normalization network layer, a PReLU activation layer, a one-dimensional convolution layer, and a batch normalization network layer to form the first data path. The 1×1 convolution network is used as the residual path. The results of the first data path and the residual path are added, and the processing result is obtained through the PReLU activation layer.
[0051] Step S4: inputting the acoustic coding features into a multi-path sequence modeling network, modeling the temporal dependency of the acoustic coding features, and extracting speech acoustic features;
[0052] In this embodiment, the multi-path sequence modeling network is within the sequence modeling model;
[0053] The multi-path sequence modeling network may adopt, but is not limited to, a dual-path recurrent neural network, a gated dual-path recurrent neural network, or a dual-path Transformer. This embodiment uses a gated dual-path recurrent neural network as the basic network structure of the sequence modeling model.
[0054] The sequence modeling model includes layer normalization (Layer Norm), 1×1 convolutional network, segmentation layer and 4-layer stacked dual-path recurrent neural network, followed by PReLU activation layer, 2D convolutional network layer, overlap-addition layer (Overlap-Add), and then divided into two data paths. The two data paths are 1×1 convolutional network and Tanh activation function as gating signal, 1×1 convolutional network and Sigmoid as the signal to be processed. The gating operation of the signal to be processed is completed by multiplication, and then the speech acoustic features are obtained through 1×1 convolutional network processing.
[0055] Specifically, each layer of the dual-path RNN consists of two parts: an intra-chunk processing module and an inter-chunk processing module. The intra-chunk processing module consists of a pair of intra-chunk RNNs, which can be quickly implemented using LSTM or GRU. The intra-chunk RNN processes the input 3-D tensor along the second dimension. The outputs of the pair are then point-by-point multiplied and concatenated with the input 3-D tensor of the same size. A linear layer is then used to map the output to a tensor of the same size as the input, which is then transposed to produce the intra-chunk output 3-D tensor. The inter-chunk processing module is structurally identical to the intra-chunk processing module, differing in that both the inter-chunk RNN and the concatenation operations process along the third dimension.
[0056] Step S5. Use the obtained speech acoustic features to predict the mean opinion score (MOS) of the speech to be detected, and map the predicted normalized result to the value space of the mean opinion score through the denormalization method to obtain the final quality evaluation result of the speech to be detected.
[0057] In this embodiment, the specific steps are:
[0058] The obtained speech acoustic features are fed into a speech quality assessment model, which uses the extracted speech acoustic features to make a preliminary prediction of the mean opinion score (MOS) of the speech to be tested. The preliminary prediction result is normalized based on the attention-based pooling network. The normalized result is then mapped to the value space of the mean opinion score through the denormalization method to obtain the final quality assessment result of the speech to be tested.
[0059] Voice quality assessment models such as Figure 4 As shown in the figure, it includes an attention pooling network model and a denormalization model. The network portion of the attention pooling network model includes a feedforward network layer, a mask layer, and a softmax layer. Self-attention calculation is then performed, and the attention weights are multiplied by the speech acoustic features before passing through a fully connected layer. The output of the attention pooling network is then normalized and mapped to the mean opinion score range. The denormalization model is then used to obtain the final speech quality prediction and evaluation results.
[0060] It should be noted that the specific implementation method of the denormalization model includes but is not limited to the following principle method, and its principle formula is as follows:
[0061]
[0062] It includes the tanh activation layer and the denormalization method. The tanh calculation obtains a score between 0 and 1, and the denormalization method maps the normalized result to the value range of the subjective average opinion score. Here For the final scoring results, This is the scoring output of the attention pooling layer.
[0063] It should be noted that the present invention provides an automated speech quality assessment method without reference speech. Before implementing step S1, a speech quality assessment model is also trained. The specific steps include:
[0064] A. Obtain a training corpus dataset, including the manual MOS score results for each corpus;
[0065] Obtain a training corpus dataset with human scoring and quality evaluation. This dataset includes data collected from different devices, signal-processed data, and speech synthesis data.
[0066] B. Preprocess the training corpus data set and extract the spectral features of the corpus data;
[0067] Spectral features are extracted from all corpus data in the training corpus dataset. Mel-spectrogram features are used as spectral features. It should be noted that in addition to Mel-spectrogram as spectral features, methods such as wav2vector can also be used as spectral features of the speech quality assessment model.
[0068] C. Send the extracted spectral features to the speech acoustic encoder to extract speech acoustic coding feature information;
[0069] The extracted acoustic coding features are fed into an automated no-reference speech quality assessment model. It should be noted that the mean square error is used as a loss function, but is not limited to it, to perform reverse updates on the model during the training process.
[0070] D. Feed the speech acoustic coding information into the sequence modeling model to extract the speech acoustic features;
[0071] E. Feed the extracted acoustic features into the speech quality assessment model to obtain the speech quality assessment model's assessment results. This result is then compared with the manual scoring results in the original training set. Based on the resulting error, the parameters in the speech quality assessment model and acoustic feature extraction model are updated. Ultimately, the trained speech acoustic feature extraction model and speech quality assessment model are obtained.
[0072] The present embodiment provides an automated speech quality assessment method without reference speech, which can detect the human hearing quality score of speech audio in real time to evaluate the audio transmission quality. At the same time, the synthesized speech can be evaluated for human hearing, and the quality of speech synthesis data can be obtained conveniently and at low cost. It can be used for speech synthesis model or speech enhancement model, speech extraction model performance evaluation and speech synthesis model or speech enhancement, speech extraction model training guidance, which greatly reduces the human cost and time cost of human hearing speech quality assessment. At the same time, the denormalization module performs a derivable mapping between the human subjective scoring range and the output range of the deep neural network model, which not only limits the scoring output range of the neural network to a certain value range, but also avoids the problem of inaccurate scoring estimation caused by forced clipping of the model output due to the result being greater than or less than the maximum or minimum value.
[0073] Example 2:
[0074] An automated speech quality assessment system for speech without reference, comprising a speech collection module, an acoustic feature extraction module, and a speech quality scoring module, wherein the acoustic feature extraction module comprises a spectral feature extraction module, an acoustic encoder module, and a sequence modeling module, and the speech quality scoring module comprises an attention pooling network module and an anti-normalization module;
[0075] A voice collection module is used to collect the acquired voice data to be detected in a storage device through an acoustic sensor;
[0076] The acoustic feature extraction module reads the speech data to be detected from the storage device, extracts the spectral features of the speech to be detected through the spectral feature extraction module, and then the acoustic encoder module encodes the spectral features of the speech to be detected based on the spectral features and extracts the corresponding acoustic coding features. The sequence modeling module models the time dependency of the acoustic coding features based on the acoustic coding features to obtain the acoustic features of the speech.
[0077] The speech quality scoring module includes an attention pooling network module and a denormalization module. The attention pooling network module predicts the mean opinion score of the speech acoustic features of the input speech to be detected, and maps the predicted normalized prediction result to the value space of the mean opinion score through the denormalization module to obtain the final quality evaluation result of the speech to be detected, and the evaluation result is fed back.
[0078] The present invention provides an automated speech quality assessment system for speech without reference, equipped with a device system capable of automated speech quality assessment. This system can perform real-time evaluation of existing communication systems, speech signal processing systems, or speech synthesis systems, monitor the system's speech output quality in real time, and detect the quality of the system's speech processing results under different circumstances. It can also serve as a voice data screening method during speech acquisition, quickly and effectively filtering out low-quality speech data such as high noise levels, thereby reducing speech acquisition costs.
[0079] Although the present invention is described herein with reference to illustrative embodiments of the present invention, the above embodiments are merely preferred embodiments of the present invention, and the embodiments of the present invention are not limited to the above embodiments. It should be understood that those skilled in the art can design many other modifications and implementations, which will fall within the scope and spirit of the principles disclosed in this application.
Claims
1. An automated speech quality assessment method without reference speech, characterized in that: The tool includes the following steps: Step S1. Obtain the voice data to be detected; Step S2. extracting the spectral features of the speech to be detected; Step S3. Encode the spectral features of the speech to be detected and extract the corresponding acoustic coding features; Step S4: feeding the acoustic coding features into a multi-path sequence modeling network, modeling the temporal dependency of the acoustic coding features, and extracting speech acoustic features; Step S5. Predicting the mean opinion score of the speech to be detected using the extracted speech acoustic features, and mapping the predicted normalized result to the value space of the mean opinion score through a denormalization method to obtain the final quality assessment result of the speech to be detected; The multi-path sequence modeling network is arranged in the sequence modeling model, which includes layer normalization, a 1×1 convolutional network, a segmentation layer, and a 4-layer stacked dual-path recurrent neural network in sequence, and then connected to a PReLU activation layer, a two-dimensional convolutional network layer, and an overlap-addition layer. The data is then divided into two paths, and the two paths of data are a 1×1 convolutional network and a Tanh activation function as a gating signal, and a 1×1 convolutional network and a Sigmoid as a signal to be processed. The gating operation of the signal to be processed is completed by multiplication, and then the speech acoustic features are obtained after being processed by a 1×1 convolutional network.
2. The method for automatic speech quality assessment without reference speech according to claim 1, wherein: In step S2, Mel-spectrogram features are extracted as spectral features of the speech to be detected.
3. The method for automatic speech quality assessment without reference speech according to claim 1, wherein: In step S3, the spectral features of the speech to be detected are encoded using an acoustic encoder module, which is composed of a one-dimensional convolutional network and a residual network.
4. The method for automatic speech quality assessment without reference speech according to claim 3, wherein: The one-dimensional convolutional network of the acoustic encoder module has an input dimension of 32 and an output dimension of 128; The residual network consists of three residual network blocks. The parameters of the three residual network blocks are: the input dimension of the first residual network block is 128, and the output dimension is 128; the input dimension of the second residual network block is 128, and the output dimension is 256; the input dimension of the third residual network block is 256, and the output dimension is 256; each residual network block is composed of a one-dimensional convolution layer, a batch normalization network layer, a PReLU activation layer, a one-dimensional convolution layer, and a batch normalization network layer to form the first data path. The 1×1 convolution network is used as the residual path. The results of the first data path and the residual path are added, and the encoding result is obtained through the PReLU activation layer.
5. The method for automatic speech quality assessment without reference speech according to claim 1, wherein: The multi-path sequence modeling network is a dual-path recurrent neural network, a gate-based dual-path recurrent neural network or a dual-path Transformer.
6. The method for automatic speech quality assessment without reference speech according to claim 5, characterized in that: Each layer of the dual-path recurrent neural network includes an intra-block processing module and an inter-block processing module. The intra-block processing module consists of a pair of intra-block recurrent neural networks, which are quickly implemented using LSTM or GRU. The intra-block recurrent neural network processes along the second dimension of the input three-dimensional tensor. The output of the intra-block recurrent neural network is then point-by-point multiplied and concatenated with the input three-dimensional tensor of the same size. A linear layer is then used to map the output to a tensor of the same size as the input, which is then transposed to obtain the output three-dimensional tensor of the intra-block processing. The inter-block processing module is structurally the same as the intra-block processing module. The inter-block recurrent neural network and splicing operations of the inter-block processing module are both processed along the third dimension.
7. The method for automatic speech quality assessment without reference speech according to claim 1, wherein: The step S5 is implemented by a speech quality assessment model, which includes an attention pooling network model and a denormalization model. The specific method is: The extracted speech acoustic features pass through the network part of the attention pooling network model and then perform self-attention calculation. The attention weight is multiplied by the speech acoustic features and then passes through the fully connected layer. The output obtained after normalization of the attention pooling network model is mapped to the scoring range of the average opinion score. The final quality prediction evaluation result of the speech to be tested is obtained through the denormalization model.
8. The method for automatic speech quality assessment without reference speech according to claim 7, characterized in that: The denormalization model is implemented according to the following formula: ； It includes the tanh activation layer and the denormalization method. The tanh calculation obtains a score between 0 and 1, and then the denormalization method maps the normalized result to the value range of the subjective average opinion score. Here For the final scoring results, This is the scoring output of the attention pooling layer.
9. An automated speech quality assessment system without reference speech, for implementing the automated speech quality assessment method without reference speech as claimed in any one of claims 1 to 8, characterized in that: It includes a speech collection module, an acoustic feature extraction module and a speech quality scoring module, wherein the acoustic feature extraction module includes a spectrum feature extraction module, an acoustic encoder module and a sequence modeling module; A voice collection module is used to collect the acquired voice data to be detected in a storage device through an acoustic sensor; The acoustic feature extraction module reads the speech data to be detected from the storage device, extracts the spectral features of the speech to be detected through the spectral feature extraction module, encodes the spectral features of the speech to be detected based on the spectral features and extracts the corresponding acoustic coding features, and the sequence modeling module models the time dependency of the acoustic coding features based on the acoustic coding features to obtain the acoustic features of the speech; The speech quality scoring module includes an attention pooling network module and a denormalization module. The attention pooling network module predicts the mean opinion score of the speech acoustic features of the input speech to be detected, and maps the predicted normalized prediction results to the value space of the mean opinion score through the denormalization module to obtain the final quality assessment result of the speech to be detected.