A neural network-based end-to-end channel quality assessment method and system

Abstract

The application discloses an end-to-end channel quality evaluation method based on a neural network, which comprises the following steps: collecting far-field noisy speech; adjusting a speech quality evaluation system to a first quality evaluation mode according to an acoustic environment in which the far-field noisy speech is located; the speech quality evaluation system supports multiple quality evaluation modes, including the first quality evaluation mode; inputting the far-field noisy speech into the adjusted speech quality evaluation system, and using a mapping function to infer a sentence-level quality score of the far-field noisy speech under the first quality evaluation mode, which is taken as a quality evaluation result of the far-field noisy speech under the first quality evaluation mode; and the speech quality evaluation system is obtained by training a neural network technology. The application predicts a score for speech quality evaluation by using a non-invasive method, can dynamically model different quality evaluation modes, does not need to collect corresponding reference speech, improves evaluation efficiency, and can quickly deploy the speech quality evaluation system in various evaluation environments.

Description

Technical Field

[0001] The present invention relates to the field of speech quality assessment, and in particular to a neural network-based end-to-end channel quality assessment method and system. Background Art

[0002] Speech quality is a subjective opinion based on how the listener perceives the speech they hear. Therefore, objectively evaluating speech quality is challenging, especially when a clean reference speech is not available (also known as non-intrusive or single-ended speech quality assessment). Although perceptual evaluation of speech quality (PESQ) is widely used to assess speech quality in industrial applications, the need for a clean reference speech significantly limits the applicability of this assessment tool in real-world scenarios.

[0003] Although ITU-T has published Recommendation P.563 as its standard algorithm for non-intrusive objective speech quality assessment, it is designed for 3.1-kHz (narrowband) telephony applications. Several non-intrusive speech quality assessment models have also been proposed. Sharma et al. used a classification and regression tree (CART) to predict quality scores based on a number of handcrafted features. Soni et al. applied a subband autoencoder to initially extract features for use in the subsequent neural network-based prediction model. Although these methods have achieved good prediction results, the features used for prediction (most of which are complex handcrafted features) are not jointly optimized with the back-end evaluation model (not end-to-end).

[0004] In recent years, deep learning has demonstrated its powerful ability to learn mapping functions in many different applications. Therefore, the present invention uses neural network modeling to perform non-invasive speech quality assessment, which is applicable to a variety of assessment environments. Summary of the Invention

[0005] To address the deficiencies in the prior art, embodiments of the present invention provide a neural network-based end-to-end channel quality assessment method and system, which can address the problem of assessing speech quality in an environment without reference speech, as well as the problem of rapidly deploying speech quality assessment systems in multiple assessment environments.

[0006] In a first aspect, an embodiment of the present invention describes a neural network-based end-to-end channel quality assessment method, the method comprising the following steps:

[0007] Collect far-field noisy speech;

[0008] adjusting the speech quality assessment system to a first quality assessment mode according to the acoustic environment in which the far-field noisy speech is located; wherein the speech quality assessment system supports multiple quality assessment modes including the first quality assessment mode;

[0009] The far-field noisy speech is input into the adjusted speech quality assessment system, and the speech quality assessment system uses a mapping function to infer the sentence-level quality score of the far-field noisy speech under the first quality assessment method; the sentence-level quality score is used as the quality assessment result of the far-field noisy speech under the first quality assessment method; the speech quality assessment system is trained by neural network technology.

[0010] Preferably, the speech quality assessment system includes an analysis filter group and a quality assessment backbone network; the analysis filter group is used to perform short-time Fourier transform on the collected far-field noisy speech to obtain the spectrum signal of the far-field noisy speech; the quality assessment backbone network is used to perform quality assessment on the spectrum signal of the far-field noisy speech.

[0011] Preferably, the acoustic environment includes noise reduction, dereverberation, and anti-interference; the quality assessment method includes objective speech quality assessment PESQ and short-time target clarity STOI.

[0012] In a second aspect, an embodiment of the present invention describes a training method for a speech quality assessment system, comprising:

[0013] Collect far-field noisy speech and corresponding clean reference speech;

[0014] Selecting a first quality assessment method, comparing the far-field noisy speech with a corresponding clean reference speech, and obtaining a quality assessment label for the far-field noisy speech under the first quality assessment method;

[0015] Adjusting the speech quality assessment system to a first quality assessment mode; wherein the speech quality assessment system supports multiple quality assessment modes including the first quality assessment mode;

[0016] Inputting the far-field noisy speech and the quality assessment label of the far-field noisy speech in the first quality assessment mode into the speech quality assessment system; the speech quality assessment system uses a mapping function to infer the frame-level quality score and sentence-level quality score of the far-field noisy speech in the first quality assessment mode; and using the sentence-level quality score as the quality assessment result of the far-field noisy speech in the first quality assessment mode;

[0017] An overall loss function is formed based on the frame-level quality score, the sentence-level quality score and the quality assessment label of the far-field noisy speech under the first quality assessment method; and the neural network parameters of the speech quality assessment system are adjusted according to the evaluation result of the overall loss function.

[0018] Preferably, the quality assessment method is determined according to the business objectives; the business objectives are dynamically confirmed according to different acoustic environments; the acoustic environment includes noise reduction, dereverberation, and anti-interference; the quality assessment method includes objective speech quality assessment PESQ and short-time target clarity STOI.

[0019] Preferably, when the evaluation result meets a set threshold condition, the training is stopped to obtain the trained speech quality evaluation system.

[0020] In a third aspect, an embodiment of the present invention describes an end-to-end channel quality assessment system based on a neural network, comprising:

[0021] Acquisition module, used to collect far-field noisy speech;

[0022] An adjustment module, configured to determine a quality assessment method according to a business objective, and adjust the number of neural network submodules of a quality assessment backbone network according to the quality assessment method;

[0023] An analysis filter bank is used to perform a short-time Fourier transform on the collected far-field noisy speech to obtain a spectrum signal of the far-field noisy speech;

[0024] The quality assessment backbone network is used to perform quality assessment on the spectrum signal of the far-field noisy speech.

[0025] Preferably, the business objectives are dynamically confirmed according to different acoustic environments; the acoustic environments include noise reduction and dereverberation; and the quality assessment methods include objective speech quality assessment PESQ and short-time target clarity STOI.

[0026] The present invention provides an end-to-end channel quality assessment method and system based on a convolutional dual recurrent neural network. The method uses a non-invasive method to predict scores for speech quality assessment, can dynamically model different quality assessment methods, and does not require the collection of corresponding reference speech, thereby improving assessment efficiency and enabling rapid deployment of speech assessment systems in a variety of assessment environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without making any creative work.

[0028] Figure 1 A schematic diagram of a flow chart of a neural network-based end-to-end channel quality assessment method provided by an embodiment of the present invention;

[0029] Figure 2 A structural framework diagram of an end-to-end channel quality assessment system based on a neural network provided by an embodiment of the present invention;

[0030] Figure 3 A flowchart of a training method for a speech quality assessment system provided by an embodiment of the present invention;

[0031] Figure 4 A schematic diagram of the structure of a voice quality assessment system provided by an embodiment of the present invention;

[0032] Figure 5 A schematic diagram of a neural network submodule based on a dual recurrent neural network provided in an embodiment of the present invention;

[0033] Figure 6 Schematic diagram of the score prediction network module provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0035] Figure 1 A flowchart of a neural network-based end-to-end channel quality assessment method provided in an embodiment of the present invention includes the following steps:

[0036] Step S11: collecting far-field noisy speech.

[0037] Step S12: adjusting the speech quality assessment system to a first quality assessment mode according to the acoustic environment of the far-field noisy speech; the speech quality assessment system supports multiple quality assessment modes including the first quality assessment mode.

[0038] The quality assessment method is determined according to the business objectives, and the number of neural network sub-modules of the quality assessment backbone network in the speech quality assessment system is adjusted according to the quality assessment method; the business objectives are dynamically confirmed according to different acoustic environments; the number of neural network sub-modules corresponds to the assessment complexity of the quality assessment method.

[0039] The speech quality assessment system includes an analysis filter bank and a quality assessment backbone network. The analysis filter bank is used to perform short-time Fourier transform on the collected far-field noisy speech to obtain the spectrum signal of the far-field noisy speech; the quality assessment backbone network is used to perform quality assessment on the spectrum signal of the far-field noisy speech.

[0040] In actual operation, the more the number M of neural network sub-modules of the quality assessment backbone network is, the higher the evaluation complexity of the corresponding quality assessment method is, where M∈[1,N], N is the number of neural network sub-modules set during training.

[0041] In one embodiment, the dynamic confirmation of the acoustic environment of the service target includes noise reduction, dereverberation, etc.; the quality assessment method includes objective speech quality assessment PESQ, short-time target clarity STOI, etc.

[0042] Step S13: Input the far-field noisy speech into the adjusted speech quality assessment system. The speech quality assessment system uses a mapping function to infer the sentence-level quality score of the far-field noisy speech under the first quality assessment method; and uses the sentence-level quality score as the quality assessment result of the far-field noisy speech under the first quality assessment method. The speech quality assessment system is trained by neural network technology.

[0043] Figure 2 The structural framework diagram of a neural network-based end-to-end channel quality assessment system provided in an embodiment of the present invention includes: an acquisition module 21, an adjustment module 22, an analysis filter bank 23, and a quality assessment backbone network 24. Specifically:

[0044] The acquisition module 21 is used to acquire far-field noisy speech.

[0045] The adjustment module 22 is used to adjust the speech quality assessment system to the first quality assessment mode according to the acoustic environment in which the far-field noisy speech is located, and to adjust the number of neural network submodules in the speech quality assessment system according to the computing resources of the acoustic environment.

[0046] The analysis filter bank 23 is used to perform short-time Fourier transform on the collected far-field noisy speech to obtain the spectrum signal of the far-field noisy speech.

[0047] The quality assessment backbone network 24 is used to perform quality assessment on the spectrum signal of the far-field noisy speech.

[0048] In actual operation, the more the number M of neural network sub-modules is, the higher the evaluation complexity of the corresponding quality assessment method is, where M∈[1,N], N is the number of neural network sub-modules set during training.

[0049] In one embodiment, the service objectives are dynamically determined based on different acoustic environments; the acoustic environments include noise reduction, dereverberation, etc.; and the quality assessment methods include objective speech quality assessment PESQ, short-time target clarity STOI, etc.

[0050] Figure 3 This is a flow chart of a method for training a speech quality assessment system according to an embodiment of the present invention. In the speech quality assessment system provided in an embodiment of the present invention, the speech quality assessment system includes an analysis filter bank and a quality assessment backbone network. The process for training the speech quality assessment system includes the following steps S31-S35:

[0051] Step S31 : collecting far-field noisy speech and corresponding clean reference speech.

[0052] Step S32: Select a first quality assessment method, compare the far-field noisy speech with the corresponding clean reference speech, and obtain a quality assessment label of the far-field noisy speech under the first quality assessment method.

[0053] Step S33: adjusting the speech quality assessment system to the first quality assessment mode; the speech quality assessment system supports multiple quality assessment modes including the first quality assessment mode.

[0054] Step S34: input the quality assessment labels of the far-field noisy speech and the far-field noisy speech in the first quality assessment mode into the speech quality assessment system; the speech quality assessment system uses a mapping function to infer the frame-level quality score and sentence-level quality score of the far-field noisy speech in the first quality assessment mode; and uses the sentence-level quality score as the quality assessment result of the far-field noisy speech in the first quality assessment mode.

[0055] The speech quality assessment system includes an analysis filter bank and a quality assessment backbone network, and its structure is shown in the figure below. Figure 4 shown.

[0056] The analysis filter bank uses short-time Fourier transform. The frame length is 512 and the frame shift is 128. The FFT size is also 512, which is equivalent to the frame length. The input is noisy speech. The output is the Fourier spectrum Where L is the number of speech sampling points, F is the number of Fourier frequency points (256), and T is the number of frames.

[0057] The quality assessment backbone network consists of a convolutional network encoder, a dual recurrent neural network, and a score prediction network. The convolutional network encoder consists of three complex convolutional layers. The number of channels in each convolutional layer is 64, the convolution kernel size is (5, 2), and the step size is (2, 1), (2, 1), (1, 1). The input of the convolutional network encoder is the original Fourier spectrum. The output is C = 64-dimensional feature sequence Where C=64, F′=256 / 4=64.

[0058] The dual recurrent neural network consists of a time domain recurrent neural network and a frequency domain recurrent neural network. Its structure is shown in the figure below. Figure 5 As shown, the dual recurrent neural network is modeled along the time axis and frequency axis respectively. The input of the time domain recurrent neural network is the information of each subband in H. The input of the frequency domain recurrent neural network is the information at each time point in H. The score prediction network consists of a fully connected layer and a global mean layer, and its structure is shown in the following figure: Figure 6 As shown. The input of the fully connected layer is the output of the dual recurrent neural network Output is frame-level quality score The input of the global mean layer is the frame-level quality score Output is sentence-level speech quality

[0059] Step S35: constructing an overall loss function based on the frame-level quality score, the sentence-level quality score, and the quality assessment label of the far-field noisy speech in the first quality assessment mode; and adjusting the neural network parameters of the speech quality assessment system based on the evaluation result of the overall loss function.

[0060] Using time domain loss and frequency domain loss as objective functions, we train and design an end-to-end channel quality assessment system. The specific formula is as follows:

[0061] L=L frame +L utt ,

[0062] in, L utt =(S u -S t ) 2 . L is the overall loss function, L frame is the frame-level loss function, L utt is the sentence-level loss function, S t is the score label.

[0063] The overall loss function is evaluated and the neural network parameters of the speech quality assessment system are adjusted according to the evaluation results.

[0064] In actual operation, the quality assessment method is determined according to the business objectives, and the business objectives are dynamically confirmed according to different acoustic environments.

[0065] In one embodiment, the acoustic environment includes noise reduction, dereverberation, etc.; the quality assessment method includes objective speech quality assessment PESQ, short-time target clarity STOI, etc.

[0066] When the evaluation result meets the set threshold condition, the training is stopped and a trained speech quality evaluation system is obtained.

[0067] The present invention provides an end-to-end channel quality assessment method and system based on a convolutional dual recurrent neural network. The method uses a non-invasive method to predict scores for speech quality assessment, can dynamically model different quality assessment methods, and does not require the collection of corresponding reference speech, thereby improving assessment efficiency and enabling rapid deployment of speech assessment systems in a variety of assessment environments.

[0068] It should be noted that the method provided herein is not inherently related to any particular computer, virtual device, or other device. Various general-purpose devices may also be used in conjunction with the teachings herein. Based on the above description, it is apparent that the structure required for constructing such devices is suitable. In addition, the present invention is not directed to any particular programming language. It should be understood that various programming languages ​​may be utilized to implement the present invention described herein, and the above description of the invocation of specific languages ​​and system function modules is merely for the purpose of disclosing the best mode of implementation of the invention.

[0069] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present invention may be implemented without these specific details. In some examples, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0070] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the present invention and its equivalents, the present invention is intended to include such modifications and variations.

Claims

1. A neural network-based end-to-end channel quality assessment method, characterized in that: include: Collect far-field noisy speech; adjusting the speech quality assessment system to a first quality assessment mode according to the acoustic environment in which the far-field noisy speech is located; wherein the speech quality assessment system supports multiple quality assessment modes including the first quality assessment mode; The far-field noisy speech is input into the adjusted speech quality assessment system, where the speech quality assessment system uses a mapping function to infer a sentence-level quality score of the far-field noisy speech under the first quality assessment mode; the sentence-level quality score is used as a quality assessment result of the far-field noisy speech under the first quality assessment mode; the speech quality assessment system includes a quality assessment backbone network; the quality assessment backbone network is used to perform quality assessment on a spectral signal of the far-field noisy speech; The quality assessment backbone network includes a convolutional network encoder, a dual recurrent neural network, and a score prediction network; the convolutional network encoder outputs a high-dimensional representation based on the spectrum signal of the far-field noisy speech; the dual recurrent neural network includes a time-domain recurrent neural network and a frequency-domain recurrent neural network, which are respectively used to model along the time axis and frequency axis of the high-dimensional representation; the score prediction network includes a fully connected layer and a global mean layer, the fully connected layer is used to output a frame-level quality score based on the high-dimensional representation output by the dual recurrent neural network, and the global mean layer is used to output a sentence-level speech quality based on the frame-level quality score; the sentence-level speech quality is the quality assessment result of the far-field noisy speech under the first quality assessment method; the speech quality assessment system is trained by neural network technology; The neural network parameters of the speech quality assessment system are adjusted according to the evaluation results of the overall loss function; the overall loss function includes a frame-level quality score, a sentence-level quality score, and a quality assessment label of far-field noisy speech under the first quality assessment method.

2. The method according to claim 1, characterized in that The speech quality assessment system includes an analysis filter bank; the analysis filter bank is used to perform short-time Fourier transform on the collected far-field noisy speech to obtain a spectrum signal of the far-field noisy speech.

3. The method according to claim 1, characterized in that The acoustic environment includes noise reduction, dereverberation, and anti-interference; the quality assessment method includes objective speech quality assessment PESQ and short-time target clarity STOI.

4. A training method for a speech quality assessment system, characterized in that: include: Collect far-field noisy speech and corresponding clean reference speech; Selecting a first quality assessment method, comparing the far-field noisy speech with a corresponding clean reference speech, and obtaining a quality assessment label for the far-field noisy speech under the first quality assessment method; Adjusting the speech quality assessment system to a first quality assessment mode; wherein the speech quality assessment system supports multiple quality assessment modes including the first quality assessment mode; Inputting the far-field noisy speech and the quality assessment label of the far-field noisy speech in the first quality assessment mode into the speech quality assessment system; the speech quality assessment system uses a mapping function to infer the frame-level quality score and sentence-level quality score of the far-field noisy speech in the first quality assessment mode; and using the sentence-level quality score as the quality assessment result of the far-field noisy speech in the first quality assessment mode; The speech quality assessment system includes a quality assessment backbone network; the quality assessment backbone network is used to perform quality assessment on the spectrum signal of the far-field noisy speech; The quality assessment backbone network includes a convolutional network encoder, a dual recurrent neural network, and a score prediction network; the convolutional network encoder outputs a high-dimensional representation based on the spectrum signal of the far-field noisy speech; the dual recurrent neural network includes a time-domain recurrent neural network and a frequency-domain recurrent neural network, which are respectively used to model along the time axis and frequency axis of the high-dimensional representation; the score prediction network includes a fully connected layer and a global mean layer, the fully connected layer is used to output a frame-level quality score based on the high-dimensional representation output by the dual recurrent neural network, and the global mean layer is used to output a sentence-level speech quality based on the frame-level quality score; the sentence-level speech quality is the quality assessment result of the far-field noisy speech under the first quality assessment method; An overall loss function is formed based on the frame-level quality score, the sentence-level quality score and the quality assessment label of the far-field noisy speech under the first quality assessment method; and the neural network parameters of the speech quality assessment system are adjusted according to the evaluation result of the overall loss function.

5. The method according to claim 4, characterized in that The acoustic environment of the far-field noisy speech includes noise reduction, dereverberation, and anti-interference; the quality assessment method includes objective speech quality assessment PESQ and short-time target clarity STOI.

6. The method according to claim 4, characterized in that When the evaluation result of the overall loss function meets the set threshold condition, the training is stopped to obtain the trained speech quality assessment system.

7. A neural network-based end-to-end channel quality assessment system, characterized in that: include: Acquisition module, used to collect far-field noisy speech; an adjustment module, configured to adjust the speech quality assessment system to a first quality assessment mode according to the acoustic environment in which the far-field noisy speech is located, and to adjust the number of neural network submodules in the speech quality assessment system according to computing resources of the acoustic environment; An analysis filter bank is used to perform a short-time Fourier transform on the collected far-field noisy speech to obtain a spectrum signal of the far-field noisy speech; A quality assessment backbone network is used to perform quality assessment on the spectral signal of the far-field noisy speech; the quality assessment backbone network includes a convolutional network encoder, a dual recurrent neural network, and a score prediction network; the convolutional network encoder outputs a high-dimensional representation based on the spectral signal of the far-field noisy speech; the dual recurrent neural network includes a time-domain recurrent neural network and a frequency-domain recurrent neural network, which are respectively used to model along the time axis and frequency axis of the high-dimensional representation; the score prediction network includes a fully connected layer and a global mean layer, the fully connected layer is used to output a frame-level quality score based on the high-dimensional representation output by the dual recurrent neural network, and the global mean layer is used to output a sentence-level speech quality based on the frame-level quality score; the sentence-level speech quality is the quality assessment result of the far-field noisy speech under the first quality assessment method.

8. The system according to claim 7, characterized in that The acoustic environment includes noise reduction, dereverberation, and anti-interference; the quality assessment method includes objective speech quality assessment PESQ and short-time target clarity STOI.

Images