A method for learning audio quality metrics that combine labeled and unlabeled data

Abstract

A method for training a neural network-based system for determining an audio quality metric for an audio input is described. The method includes receiving as input at least one training set including audio samples. The audio samples include audio samples of a first type and audio samples of a second type, each of the first type being labeled with information representing a respective predetermined audio quality metric, and each of the second type being labeled with information representing the respective audio quality metric relative to that of a reference audio sample. The method further includes inputting the training set into the neural network-based system; and iteratively training the system to predict label information for each of the audio samples in the training set.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority from the following earlier applications, which are incorporated herein by reference: ES Application No. 202030605 (Reference No. D20045ES), filed June 22, 2020, U.S. Provisional Application No. 63 / 072,787 (Reference No. D20045USP1), filed August 31, 2020, U.S. Provisional Application No. 63 / 090,919 (Reference No. D20045USP2), filed October 13, 2020, and EP Application No. 20203277.7 (Reference No. D20045EP), filed October 22, 2020.

[0002] Technical Field The present disclosure relates generally to the field of audio processing. In particular, the present disclosure relates to techniques for speech / audio quality assessment using machine learning models or systems, and a framework for training a machine learning model or system for speech / audio quality assessment. [Background technology]

[0003] Speech and audio quality assessment is crucial for a myriad of research topics and real-world applications. The needs range from algorithm evaluation and development to fundamental analysis and informed decision making. Broadly speaking, the assessment of audio quality can be performed by subjective listening tests or by objective quality metrics. Objective metrics that correlate well with human judgments open the possibility of scaling up automated quality assessment with consistent results at a negligible fraction of the effort, time, and cost of their subjective counterparts. Traditional objective metrics rely on standard signal processing blocks such as short-time Fourier transforms or perceptually motivated blocks such as Gammatone filter banks. They, together with further processing blocks, often yield intricate and complex rule-based systems. An alternative approach is to learn speech quality directly from raw data by combining machine learning techniques with carefully selected factors and corresponding human assessments. Rule-based systems have the advantage of being perceptually motivated and to some extent interpretable, but often give a narrow focus on a particular type of signal or impairment, such as telephone signals or voice-over-IP (VoIP) degradations. Learning-based systems, on the other hand, usually require a significant amount of human-annotated data, which can be easily reused for other tasks and impairments. Both rule-based and learning-based systems can further suffer from a lack of generalization capabilities and therefore may perform poorly on samples outside the focused data.

[0004] There is therefore a need for methods and systems for performing (automatic) audio quality assessment, as well as methods for training such systems that can achieve improved performance and / or efficiency (e.g., in terms of error rate, consistency, etc.), while at the same time allowing good generalization to new audio (e.g., recordings) and / or listeners. Summary of the Invention

[0005] In view of the above, the present disclosure generally provides a method for training a neural network based system for determining an indication of audio quality of an audio input, a neural network based system for determining an indication of audio quality of input audio samples, and a method for operating a neural network based system for determining an indication of audio quality of input audio samples, as well as corresponding programs, computer readable storage media, and apparatus having the features of the respective independent claims. The dependent claims relate to preferred embodiments.

[0006] According to one aspect of the present disclosure, a method for training a deep learning based (e.g., neural network based) system for determining an audio quality metric of an audio input is provided. Training may refer to determining parameters of a deep learning model (e.g., neural network) used to implement the system. Furthermore, training may refer to iterative training. The audio quality metric of the audio input may be, for example, a score. The score may be normalized (bounded) to a predefined scale, such as between 1 and 5, as appropriate. The method may include obtaining as input at least one training set including audio samples. In particular, the audio samples may include audio samples of a first type and audio samples of a second type. More particularly, each of the first type audio samples may be labeled with information representative of a respective predefined audio quality metric (e.g., between 1 and 5) and each of the second type audio samples may be labeled with information representative of a respective audio quality metric relative to that of a reference audio sample (e.g., relative to that of another audio sample in the training set). In other words, the first type of audio samples may be understood as each including label information representing an absolute audio quality metric (e.g., normalized between 1 and 5, with 5 having the highest audio quality). In contrast, the second type of audio samples may be understood as each including label information indicating a relative audio quality metric. As will be understood and appreciated by those skilled in the art, the reference audio sample used herein may be, but is not necessarily required to be, another audio sample in the training set. In other words, the reference audio sample may be an external reference audio sample (i.e., one not in the training set) or an internal reference audio sample (i.e., one in the training set).Further, the reference audio sample may be any suitable audio sample, e.g., a predefined or predetermined one that can be used to serve as a reference (in a comparison), such that, in a broad sense, a relative metric can be determined (e.g., calculated) by comparing the audio sample to the reference audio sample. In some examples, the relative label information may include information indicating that the audio sample is more degraded (or less degraded) than a (predetermined) reference audio sample (e.g., another audio sample in a training set). In some examples, the relative label information may include information representing, for example, a particular degradation function (and optionally a corresponding degradation strength) that has been applied to the reference audio sample (e.g., another audio sample in a training set) when generating the (degraded) audio sample. Of course, any other suitable relative label information may be included as necessary or appropriate, as will be understood and appreciated by those skilled in the art. The method may further include inputting the training set into a deep learning-based system and iteratively training the system to predict label information for each of the audio samples in the training set. Training may be based on multiple loss functions. In particular, multiple loss functions may be generated that reflect the difference between the label information of the audio samples in the training set and their respective predictions.

[0007] Organized as described above, in the broadest sense, the proposed method is able to train a neural network that leads to a non-intrusive quality assessment. Since the assessment is learned from the data, the focus can be re-set by changing the audio type on which it is trained, and it is also possible to choose the degradation to learn from. In particular, the proposed method is generally semi-supervised, which means that it can take advantage of both absolute and relative assessments from various data sources. In this way, the need for expensive and time-consuming listener data can be mitigated. In addition to learning from multiple sources, the proposed method can learn from multiple features of those sources by training the network based on multiple loss functions (generated according to the audio samples in the data sources), thereby leading to a much more general automatic assessment.

[0008] In some examples, the first type of audio samples may include human-annotated audio samples, each of which may be labeled with information representative of a respective predetermined audio quality metric. As will be understood and appreciated by one of ordinary skill in the art, the audio samples may be annotated by any suitable means, for example, by audio experts, ordinary listeners, mechanical turkers (e.g., crowd-sourcing), etc.

[0009] In some examples, the human-annotated audio samples may include mean opinion score (MOS) audio samples and / or just-noticeable difference (JND) audio samples. Some possible examples of MOS and JND data sets are provided in Sections B.1 and B.2, respectively, of the appendix included herein.

[0010] In some examples, the second type of audio samples includes algorithmically (or programmatically, artificially) generated audio samples, each of which is labeled with information representing a relative predetermined audio quality metric.

[0011] In some examples, each of the algorithmically generated samples may be generated by selectively applying at least one degradation function, each with a respective degradation strength, to the reference audio sample or to another algorithmically generated audio sample. In such examples, the label information may include information indicating each degradation function and / or each degradation strength that has been applied. Of course, any other suitable algorithm and / or program may be used to generate the second type of audio sample, as will be appreciated by those skilled in the art.

[0012] In some examples, the label information may further include information representing degradation relative to one another, i.e., in some examples, the label information may further include information representing degradation relative to other audio samples in the training set or relative to a reference audio sample, for example, the label information may include relative information indicating that an audio sample is relatively more or less degraded than another audio sample (e.g., another audio sample in the training set or an external reference audio sample).

[0013] In some examples, the degradation function may be selected from among multiple available degradation functions, which may be implemented as a pool of degradation functions. Additionally or alternatively, each degradation strength may be set (e.g., by an expert, a listener, or an author) such that at its minimum value, the degradation is still perceptually noticeable.

[0014] In some examples, the available impairment functions may include functions, treatments, or process-related functions of one or more of reverberation, clipping, encoding with different codecs, phase distortion, audio reversal, and background noise. Furthermore, the (background) noise may include real (e.g., recorded) background noise or artificially generated background noise. Note that in some cases, the impairment strength selected may be only one aspect of all impairments, and in other related aspects, it may be randomly sampled among empirically selected values. For example, in the case of a reverb effect, the signal-to-noise ratio (SNR) may be selected as the primary strength, but the reverberation type, width, delay, etc. may be selected randomly. Some possible examples of impairments and / or strengths are given in Section C of the appendix included herewith.

[0015] In some cases, the audio samples generated by the algorithm are generated for pairs of audio frames {x i ,x j} and / or quadruple audio frames {x ik ,x il ,x jk ,x jl In particular, audio frame x i may be generated by selectively applying at least one degradation function, each with a respective degradation strength, to a (e.g., external) reference audio frame (or an audio frame from a training set). Then, for an audio frame x j apply at least one degradation function, each with a respective degradation strength, to the audio frame x i , and may be generated by selectively applying ik ,x il is the time delay for each audio frame x i By selectively applying i The audio frame x may be extracted from jk ,xjl is the time delay for each audio frame x j By selectively applying j By way of example and not by way of limitation, audio frame x i may be 1.1 seconds long, and 1.1 seconds of audio frame x i Audio frame x to be extracted ik ,x il As can be understood and appreciated by those skilled in the art, audio samples may be of any suitable length depending on various implementations and / or requirements. It can be generated by any means.

[0016] In some examples, the loss function may include a first loss function representing a MOS error metric. The first loss function may be calculated based on a difference between a MOS ground truth for the audio samples in the training set and a prediction of the audio samples. In this sense, the first loss function may in some cases be considered to be indicative of a MOS opinion score metric. Of course, in addition to difference, any other suitable means may be used to determine (calculate) the first loss function (or any other suitable loss function described in more detail below), such as a suitable mathematical concept such as divergence or cross entropy, as will be understood and appreciated by those skilled in the art.

[0017] In some examples, the label information of the second type of audio samples may include relative (label) information indicating whether one audio sample is more (or in some cases less) degraded than another audio sample. In addition to or instead of the first loss function described above, the other loss function may include a second loss function representing a pairwise ranking metric. In particular, the second loss function may be calculated based on the ranking set by the label information including the relative degradation information and its prediction.

[0018] In some examples, the system may be trained in a manner that results in an audio quality metric that indicates better audio quality for one audio sample that is less degraded than another audio sample that is more degraded.

[0019] In some examples, the label information of the second type of audio samples may include relative information representing the perceptual relevance between the audio samples. The perceptual relevance may represent, for example, the perceptual difference or the perceptual similarity between two audio samples or between two pairs of audio samples. That is, roughly speaking, if two audio signals are extracted from the same (audio) source and differ by only a few audio samples, or if the difference between the two signals is perceptually irrelevant, their respective quality metrics (or quality scores) should be essentially the same. Complementarily, if two signals are perceptually distinguishable, their metric / score difference should exceed a certain margin. In particular, these two ideas can also be extended to pairs of pairs, for example by considering the consistency between the pairs of score differences. Thus, the loss function may additionally or alternatively include a third loss function representing a consistency metric, in particular the third loss function may be calculated based on the difference between the label information including the information of the perceptual relevance and its prediction. In this sense, the third loss function may in some cases be thought of as representing a score consistency metric.

[0020] In some examples, the consistency metric may indicate whether two or more audio samples have the same degradation function and / or degradation strength and correspond to the same time frame.

[0021] In some examples, the label information of the second type of audio sample may include relative information representing whether an audio sample is applied with the same degradation function and the same degradation strength as another audio sample. Thus, the loss function may additionally or alternatively include a fourth loss function representing a (same or different) degradation state metric. In particular, the fourth loss function may be calculated based on the difference between the label information including the relative degradation information / state and its prediction.

[0022] In some examples, the label information of the second type of audio samples may include relative information representing perceptual differences with respect to one another, and thus the loss function may additionally or alternatively include a fifth loss function representing a JND metric, the fifth loss function being calculated based on the difference between the label information including the relative perceptual differences and the prediction.

[0023] In some examples, the label information of the second type of audio sample may include information representative of a degradation function applied to the audio sample. Thus, the loss function may additionally or alternatively include a sixth loss function representative of a degradation type metric. In particular, the sixth loss function may be calculated based on a difference between the label information including the type information of the respective degradation function and its prediction.

[0024] In some examples, the label information of the second type of audio sample may include information representing a degradation strength applied to the audio sample, and the loss function may additionally or alternatively include a seventh loss function representing a degradation strength metric, and the seventh loss function may be calculated based on a difference between the label information including the respective degradation strength information and the prediction.

[0025] In some examples, the loss function may additionally or alternatively include an eighth loss function representing a regression metric. In particular, the regression metric may be calculated according to at least one of a reference-based quality measure and a non-reference based quality measure.

[0026] In some examples, the reference-based quality measure may include, but is not limited to, at least one of: a perceptual evaluation of speech quality (PESQ), a composite measure for signal (CSIG), a composite measure for noise (CBAK), a composite measure for overall quality (COVL), a segmental signal-to-noise ratio (SSNR), a log-likelihood ratio (LLR), a weighted slope spectral distance (WSSD), a short-term objective intelligibility (STOI), a scale-invariant signal distortion ratio (SISDR), Mel cepstral distortion, and a log-Mel-band distortion. Of course, any other suitable reference-based and / or non-reference-based quality measures may be used, as will be appreciated by those skilled in the art.

[0027] In some examples, each of the audio samples in the training set may be used in at least one of the multiple loss functions. That is, some of the audio samples in the training set may be reused or shared by one or more loss functions. For example, audio samples (generated by the algorithm) for calculating the third loss function (i.e., score consistency metric) may be reused when calculating the fourth loss function (i.e., same / different degradation condition metric), or vice versa. Thus, the efficiency of training the system may be significantly improved. In particular, the final loss function for training may be generated based on an averaging process for one or more of the multiple loss functions. As will be appreciated by those skilled in the art, any other suitable means or process may be used to generate the final loss function based on any number of suitable loss functions, depending on various implementations and / or requirements.

[0028] In some examples, the system may include an encoding stage (or simply called an encoder) that maps (e.g., transforms) the audio input into a feature space representation. The feature space representation may be, for example, a (feature) latent space. The system may include an evaluation stage that generates a prediction of label information based on the feature space representation.

[0029] In some examples, the encoding stage for generating the intermediate representation may include a neural network encoder.

[0030] In some examples, each of the multiple loss functions may be determined based on a neural network including a linear layer or a multilayer perceptron (MLP).

[0031] According to another aspect of the present disclosure, a deep learning-based (e.g., neural network-based) system for determining an audio quality indicator of an input audio sample is provided. The system can be trained according to any one of the above examples. In particular, the system can include an encoding stage and an evaluation stage. More specifically, the encoding stage can be configured to map the input audio sample to a feature space representation. Furthermore, the evaluation stage can be configured to predict information representative of a predetermined audio quality metric based on the feature space representation, and further predict information representative of a relative audio quality metric to a reference audio sample. As can be understood and appreciated by those skilled in the art, the reference audio sample used herein can be, but is not necessarily, another audio sample in a training set for training the system. In other words, the reference audio sample can be an external reference audio sample (i.e., one not in the training set) or an internal reference audio sample (i.e., one in the training set). Furthermore, the reference audio sample may be any suitable audio sample, e.g., a predefined or predetermined one that can be used to serve as a reference (for comparison), such that, in broad terms, a relative metric can be determined (e.g., calculated) by comparing the audio sample to the reference audio sample. Furthermore, the predicted information (e.g., indicative of a relative audio quality metric with respect to the reference audio sample) may be used to further train (regularize) the system.

[0032] In some examples, the system may be configured to receive at least one training set as an input. In particular, the training set may include audio samples of a first type and audio samples of a second type, each of the audio samples of the first type being labeled with information representative of a respective predefined audio quality metric, and each of the audio samples of the second type being labeled with information representative of the respective audio quality metric relative to that of a reference audio sample or relative to that of another audio sample in the training set. Furthermore, the system may be configured to input the training set into the system; and iteratively train the system based on the training set to predict label information of each of the audio samples in the training set based on a plurality of loss functions, the plurality of loss functions being generated to reflect differences between the label information of the audio samples in the training set and their respective predictions.

[0033] According to another aspect of the present disclosure, a method is provided for operating a deep learning-based (e.g., neural network-based) system for determining an audio quality measure for an input audio sample. The system may correspond to any one of the exemplary systems described above; the system may be trained according to any one of the exemplary methods described above. For example, the system may include an encoding stage and an evaluation stage. In particular, the method may include a step of mapping the input audio sample to a feature space representation by the encoding stage. The method may further include a step of predicting, by the evaluation stage, information representative of a predetermined audio quality metric and information representative of a relative audio quality metric to a reference audio sample based on the feature space representation. As will be understood and appreciated by those skilled in the art, the reference audio sample used herein may be, but is not necessarily required to be, another audio sample in the training set. In other words, the reference audio sample may be an external reference audio sample (i.e., not in the training set) or an internal reference audio sample (i.e., in the training set). Furthermore, the reference audio sample may be any suitable audio sample, e.g., a predefined or predetermined audio sample that can be used to serve as a reference (for comparison), such that, in broad terms, a relative metric can be determined (e.g., calculated) by comparing the audio sample to the reference audio sample. Furthermore, the predicted information (e.g., indicative of a relative audio quality metric with respect to the reference audio sample) may be used to further train (regularize) the system.

[0034] According to another aspect of the present disclosure, a computer program is provided that may include instructions that, when executed by a processor, cause the processor to perform all of the steps of the exemplary methods described throughout the present disclosure.

[0035] According to a further aspect, there is provided a computer readable storage medium capable of storing a computer program as described above.

[0036] According to yet another aspect, an apparatus is provided that includes a processor and a memory coupled to the processor, the processor being capable of being configured to cause the apparatus to perform all steps of the example methods described throughout this disclosure.

[0037] It will be understood that system features and method steps may be interchangeable in many ways. In particular, as will be appreciated by those skilled in the art, details of a disclosed method may be implemented by a corresponding system and vice versa. Furthermore, any description above of a method is to be understood as applying equally to a corresponding system and vice versa. [Brief description of the drawings]

[0038] Exemplary embodiments of the present disclosure will now be described with reference to the accompanying drawings. [Figure 1A] FIG. 1A is a schematic diagram of a system for audio quality assessment according to an embodiment of the present disclosure. [Figure 1B] FIG. 1B is a schematic diagram of another block diagram of a system for audio quality assessment according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a flow chart illustrating an example of a method for training a deep learning based system to determine a measure of speech quality of an audio input, according to an embodiment of the present disclosure. [Diagram 3] FIG. 3 is a flow chart illustrating an example method of operating a deep learning based system for determining an audio quality metric of an input audio sample, according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is an exemplary illustration showing various results and comparisons according to embodiments of the present disclosure. [Diagram 5]FIG. 5 is an exemplary illustration showing various results and comparisons according to embodiments of the present disclosure. [Figure 6A] FIG. 6A is an exemplary illustration showing various results and comparisons according to an embodiment of the present disclosure. [Figure 6B] FIG. 6B is an exemplary illustration showing various results and comparisons according to an embodiment of the present disclosure. [Figure 6C] FIG. 6C is an exemplary illustration showing various results and comparisons according to an embodiment of the present disclosure. [Figure 7A] FIG. 7A is an exemplary illustration showing various results and comparisons according to an embodiment of the present disclosure. [Figure 7B] FIG. 7B is an exemplary illustration showing various results and comparisons according to an embodiment of the present disclosure. [Figure 8A-8B] 8A and 8B are exemplary illustrations showing various results and comparisons according to embodiments of the present disclosure. [Figure 8C] FIG. 8C is an exemplary illustration showing various results and comparisons according to an embodiment of the present disclosure. [Figure 8D] FIG. 8D is an exemplary illustration showing various results and comparisons according to an embodiment of the present disclosure. [Figure 8E] FIG. 8E is an exemplary illustration showing various results and comparisons according to an embodiment of the present disclosure. [Figure 8F] FIG. 8F is an exemplary illustration showing various results and comparisons according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0039] It should be noted that the drawings and the following description relate to preferred embodiments by way of example only, and that from the following description, alternative embodiments of the structures and methods disclosed herein will be readily recognized as viable alternatives that can be used without departing from the principles of what is claimed.

[0040] Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying drawings. It should be noted that, wherever possible, like or similar reference numbers may be used in the drawings and may indicate like or similar functionality. The drawings depict embodiments of the disclosed systems (or methods) for illustrative purposes only. Those skilled in the art will readily appreciate from the following description that alternative embodiments of the structures and methods shown herein may be used without departing from the principles described herein.

[0041] Generally speaking, quality assessment is essential in the audio industry, with a wide range of applications from monitoring channel distortion to developing new processing algorithms. Traditionally, quality assessment is obtained from standard or professional listeners, with significant investments in money, time and infrastructure. In this disclosure, an automatic tool is proposed to provide such quality assessment.

[0042] The goal of an automatic tool (or algorithm) to measure audio quality is to obtain a reliable proxy for human evaluation, overcoming the aforementioned investments. Several automatic tools exist to measure the speech quality of audio files. Given some input audio, such tools typically produce a score between 1 and 5, which is associated with some subjective assessment of the audio quality.

[0043] One distinction between these tools is whether they use reference (clean) audio for comparison (intrusive vs. non-intrusive). Another distinction is whether they are hand-created / predefined or learned from data. Further considerations are the range of audio that is going to be analyzed and the specific degradations or distortions that the measurements can detect.

[0044] Thus, a key driver of this disclosure is the realization that additional criteria / tasks should be considered beyond correlation with conventions or measures of audio quality, such as the Mean Opinion Score (MOS), which is specifically determined to learn from such additional criteria. Another fundamental aspect of this disclosure is the recognition that there are additional problems, data sets, and tasks that can complement these criteria and help learn more robust representations of speech quality and scores.

[0045] Broadly speaking from these perspectives, the present disclosure proposes a method for training a neural network that results in an unintrusive quality assessment. Since the assessment is learned from data, the focus can be re-set by changing the audio type on which the neural network is trained, and it is also possible to select the degradations to learn from. In particular, the proposed method is generally semi-supervised, meaning that it can leverage both assessments from human listeners (e.g., incorporated in human-annotated data, often also referred to as labeled data) and raw (unrated) audio as input data (often also referred to as unlabeled data). In this way, the need for expensive and time-consuming listener data can be mitigated. In addition to learning from multiple sources, the proposed method also learns from multiple characterizations of these sources, thus leading to a much more general automated measurement. Additional design principles of the proposed method (and system) may include, but are not limited to, lightweight and fast operation, being fully differentiable in nature, and the ability to process short raw audio frames, e.g., at 48 kHz (thus resulting in time-varying dynamic estimates).

[0046] With reference to FIG. 1A, a (simplified) block diagram schematic of a system 100 for audio quality assessment according to an embodiment of the present disclosure is shown. The system 100 may be composed of an encoding stage (or simply referred to as an encoder) 1010 and an assessment stage 1020. As shown in the example of FIG. 1A, the assessment stage 1020 may include a series of "heads" 1021, 1022 and 1023, often denoted (collectively) as H. The various heads are described in detail below with reference to FIG. 1B. Roughly speaking, each head may be considered as an individual computation unit suitable for determining respective label information (e.g., absolute quality metric or relative quality metric) associated with each audio sample (frame). In general, the encoder 1010 is capable of taking raw input audio signals (e.g., audio frames) x 1000 and mapping (or transforming) them, for example, to a latent space representation (vector) z 1005. The various heads can then take these latent vectors z 1005 and compute outputs for one or more considered criteria (exemplary shown as 1025). In particular, in some cases, the pair {z i ,z j}, the head can take their concatenation (or any other suitable form) as input.

[0047] The encoder 1010 may, in some examples, be composed of four main stages, as shown in FIG. 1A. First, the encoder 1010 may transform the distribution of x 1000 by applying a μ-law formula (e.g., without quantization) using a learnable μ. In general, the μ-law algorithm (sometimes written as “mu-law”) is a companding algorithm and is primarily used, for example, in 8-bit PCM digital communication systems. In particular, the companding algorithm may be used to reduce the dynamic range of an audio signal. In analog systems, this may increase the SNR achieved during transmission, while in the digital domain, it may reduce the quantization error (thus increasing the signal-to-quantization noise ratio). For example, the value of μ may be initially initialized to 8. Next, block 1001 may be used, which in some examples may include a series of (e.g., four) pooling sub-blocks, which may consist of convolution, batch normalization (BN), rectified linear unit (ReLU) activation, BlurPool, or any other suitable block / module. By way of example and not limitation, 32, 64, 128, and 256 filters with kernel width 4 and downsampling factor 4 may be used. Of course, any other suitable implementation may be used as well, as will be appreciated by those skilled in the art. For example, possible alternatives to convolution include, but are not limited to, linear layers, recurrent neural networks, attention modules, or transformers. Possible alternatives to batch normalization include, but are not limited to, layer normalization, instance normalization, or group normalization. In some other implementations, batch normalization may be omitted entirely. Possible alternatives to ReLU include, but are not limited to, a sigmoid gate, a tanh gate, a gated linear unit, a parametric ReLU, or a leaky ReLU.Possible alternatives to BlurPool include, but are not limited to, convolution with stride, max pooling, or average pooling. It is further understood that the aforementioned alternative implementations can be combined with each other as desired or feasible, as will be appreciated by those skilled in the art.

[0048] Next, block 1002 may be used, which in some examples may include multiple (e.g., 6) residual blocks formed by BN pre-activation followed by 3 blocks (ReLU, convolution, BN). By way of example and not limitation, 512, 512, and 256 filters with kernel widths 1, 3, and 1 may be used, with residual connections by parametric linear averaging performed as follows:

number

[0049] 1B, a more detailed block diagram schematic diagram of a system 110 for audio quality assessment according to an embodiment of the present disclosure is shown. In particular, identical or similar reference numbers in the system 110 of FIG. 1B indicate identical or similar elements in the system 100 as shown in FIG. 1A, and therefore repeated description thereof may be omitted for reasons of brevity. In particular, in the exemplary system 110 of FIG. 1B, the focus is on the assessment stage 1120, where various learning / training criteria of the heads are described in detail below.

[0050] Referring to the system 110 of FIG. 1B, broadly speaking, a (convolutional) neural network can be trained that is capable of converting an audio input x 1100 into a (low-dimensional) latent space representation z 1105, the latter capable of outputting a single-valued score 1140. Similar to the one shown in FIG. 1A, the network / system may be formed by two main blocks (stages): an encoding stage (or sometimes referred to as an encoder network) 1110 that outputs a latent vector z 1105, and an evaluation stage 1120 that includes a number of different "heads" that further process the latent vector z 1105. In particular, one of the heads is in charge of generating the final score s 1140, while the remaining heads are typically used to regularize the latent space (they can also be used as predictors for the quantities being trained).

[0051] Similar to that of FIG. 1A, the encoding stage 1110 may take a μ-law logarithmic representation of the audio and pass it to a series of convolutional blocks. For example, first, multiple BlurPool blocks (e.g., 1101) may decimate the signal to a lower time span. Then, multiple ResNet blocks (e.g., 1102) may further process the obtained representation. Then, temporal statistics (e.g., 1103) such as mean, standard deviation, minimum, and maximum may be taken to summarize the audio frames. Finally, a MLP (e.g., 1104) may be used to perform a mapping between these statistics and z-values ​​1105.

[0052] Different heads can take in a vector z 1105 and predict different quantities 1121-1128. In general, at training time, every head can have a loss function imprinting desired properties onto either the score s 1140 or the latent space z 1105.

[0053] In particular, the score s may be calculated in any suitable manner, as will be appreciated by those skilled in the art. Some possible examples of how the score s may be calculated are provided, for example, in Section A of the appendix included herein.

[0054] Next, in relation to the system 110 of FIG. 1B, examples of various possible learning or evaluation criteria corresponding to possible heads and their respective loss functions are described in detail below. Some of these criteria may be considered as auxiliary tasks in some cases. In other words, not all criteria are necessarily used when performing training of the system, and some criteria may be omitted or bypassed depending on various implementations and / or requirements. Of course, as will be understood and appreciated by those skilled in the art, the criteria (or heads) are not limited to those described herein and can be extended or adapted to any particular case.

[0055] Mean Opinion Score The principle and most inherent criterion considered in conventional approaches may be the MOS error. In some cases, this may simply be referred to as the score head 1121. In general, this score head may take z 1105 as input and pass it through, for example, a linear layer (which may be an MLP or any other suitable neural network) 1131 to generate a single quality score value s. As an example, such a score may be bounded by a sigmoid function and rescaled to be, for example, between 1 and 5 (for example, 5 being the highest quality). Also, to calculate the loss of this head, for example, a rating provided by a human listener may be used, if available. An alternative may be to use a rating provided by other existing quality measures (reference-based or non-reference-based). In other words, it may be considered that, roughly speaking, the loss function may include a first loss function representing a MOS error metric, which may be calculated based on the difference between the MOS ground truth of the audio samples in the training set and the prediction of the audio samples.

[0056] More specifically (but not by way of limitation), in learning-based approaches, a supervised regression problem is typically posed as follows:

number

[0057] In one example, the system 110 calculates the score s by, for example, using a linear unit and a sigmoid activation σ. i Let z be the latent representation i It can be predicted from:

number

[0058] Pairwise Ranking Besides MOS, an intuitive but often overlooked idea in quality assessment may be pairwise ranking. In some cases, this may simply be referred to as the rank head 1122. Generally speaking, this pairwise ranking head 1122 may take as input, for example, a pair of scores, e.g., s1, s2, which may be obtained from a preceding score head after processing audio x1, x2. Then, if available, a flag (e.g., label information) signaling which audio is more (or less) degraded may be used to calculate a rank-based loss. For example, the loss may work towards s1 being lower than s2 if x1 is more degraded / corrupted than x2 (or vice versa). In other words, it can be considered that, roughly speaking, the loss function may include a second loss function representing a pairwise ranking metric, and the second loss function may be calculated based on the difference between the label information (e.g., the ranking set by the label information) and its prediction, including the relative degradation information.

[0059] More specifically (but not exclusively), under the concept of pairwise ranking, we consider a speech signal x j but the same (original "clean" or "cleaner") utterance x iIf we are programmatically (algorithmically) degraded versions of s, then their scores should reflect such a relationship, i.e., i ≧s j This idea may then be introduced into the training scheme by considering learning-to-rank strategies. In one example, this may follow the margin loss formula:

number

[0060] In one example, the pair {x i ,x j} 1142 may be generated programmatically by considering several data sets with "clean" speech (also called reference speech) and a pool of several degradation functions.

[0061] Pair of {x i ,x j} 1142 may be generated in any suitable manner. By way of example and not limitation, to form all pairs, one may proceed as follows: Uniformly sample a data set and then uniformly sample a file from it. Uniformly sample a frame of 1.1 seconds (or any other suitable length), avoiding silent or mostly silent frames. Normalize it so that its maximum absolute amplitude is 1. With probabilities 0.84, 0.12, and 0.04, we sample an impairment of 0, 1, or 2 from the pool of available impairments (more on this below). In the case of zero impairment, the signal immediately drops to x i Otherwise, the strengths for each impairment are chosen uniformly, x i These can be applied in sequence to produce Sample 1, 2, 3, or 4 impairments from the pool of available impairments with probabilities 0.75, 0.2, 0.04, and 0.01. Select the intensities uniformly and scale them by x i Apply to x in order j Generate.

[0062] It should be understood that the above implementations including the above probabilities are merely for illustrative purposes without any limitations, and any other suitable probabilities or implementations may be applied thereto, as will be understood by those skilled in the art.

[0063] Then, the generated pair {x i ,x j} may be stored together with information on the degradation type and / or intensity (eg, stored as label information).

[0064] Further information regarding possible means of generating pairs can also be found, for example, in section B.3 of the appendix contained herein.

[0065] Additionally or alternatively, random pairs may be collected, for example from (human) annotated data, e.g., by finding corresponding s * Therefore, the larger the s * may acquire index i, or vice versa. For pairs coming from annotated data, the margin constant can be, for example, α'=min(α,si * -sj * ) or any other suitable value.

[0066] Score Consistency Consistency is another overlooked concept in audio quality assessment. Generally speaking, the consistency head 1123 can take as input a pair of scores s1, s2 corresponding to audio x1, x2 respectively. Then, if available, a flag (e.g., label information) signaling whether the audios are likely to have the same degradation type and / or level can be used to compute a distance-based loss. For example, the loss may encourage s1 to move closer to s2 if x1 has the same distortion / degradation as x2 at the same level (in some cases, it may be assumed that similar original content exists in both x1, x2, if necessary). Also, the loss may be used to improve the accuracy of the loss for audios with different degradations x1, x2. ’ 1,x ’ Similar realizations x1, x2 with 2 may also be close together (e.g., x1 and x ’ 1, x2, and x ’ 2) In other words, in a broad sense, it may be considered that the loss function may include a third loss function representing a consistency metric, and that the third loss function may be calculated based on the difference between the label information including the perceptually relevant information and the prediction thereof.

[0067] More specifically (but not as a limitation), under the idea of ​​score consistency, two signals x k ,k l are extracted from (essentially) the same source and differ by only a few audio samples, or the two signals x k ,k l If the differences between s are perceptually irrelevant, then their scores should be essentially identical, i.e., s k =s l In addition, two signals x i ,x j If,x,y,i,are perceptually distinct, the difference between their scores should,be greater than a given (e.g., predetermined) margin, i.e., |s i -s j |≧β Note that these two ideas may be further extended to pairs of pairs, for example by considering the consistency between pairs of score differences. In one possible implementation, the first idea can be extended as follows: Given two signals x ik ,x jk These are x il ,k jl If it is perceptually identical to (x j x i more degraded, and signals k,l are derived from them), the score difference tends to be equal, i.e., s ik -s jk =s il -s jl It is.

[0068] In one example, taking into account all three ideas above, a consistency loss might be proposed as follows:

number

[0069] In particular, the audio frame / signal pair {x i ,x j} 1142 may be generated as described above during the computation of pairwise rankings, or in any other suitable manner. ik ,x il ,x jk ,x jl} 1142 may, for example, use a random small delay (e.g., less than 100 ms) to separate pairs x i ,x j By way of example and not by way of limitation, for a given pair {x i ,x j To form each quad from}, one may proceed as follows: Uniformly sample time delays between 0 and 100 ms. Using such delays, xi Frame x from 1s ik ,x il Extract and do the same with x j x from jk ,x jl Execute regarding.

[0070] As above, the generated quadruple {x ik ,x il ,x jk ,x jl} may be stored together with information on the degradation type and / or intensity (eg, stored as label information).

[0071] Further information regarding possible means for generating quadruplets can also be found, for example, in section B.3 of the appendix included herewith.

[0072] Additionally or alternatively, the pair {x i ,x j} and / or {x k ,x l} may be taken from a (predetermined) JND data set 1143, and the 4-tuple {x ik ,x il ,x jk ,x jl} then divides those pairs {x i ,x j} and / or {x k ,x l} may be generated.

[0073] Different / same conditions As mentioned above, L CONS It is also possible to provide information on pairs of signals corresponding to (essentially) the same degradation condition, i.e., pairs of signals undergoing the same degradation type and (optionally) the same strength, using the data generated by the program for the above. In other words, it can be considered that, in a broad sense, the loss function may include a fourth loss function representing a degradation condition metric, and the fourth loss function may be calculated based on the difference between the label information including the relative degradation information and its predicted value.

[0074] In one possible example, this information may then be included by considering the classification loss at the head 1124:

number

number

[0075] Just noticeable difference As mentioned above, if signal pairs with human annotations regarding perceptual dissimilarity (or relatedness) may be accessible or available from the training set, the notion of perceptual dissimilarity (or relatedness) may be further enforced in the latent space, for example with another classification loss in head 1125:

number

[0076] In other words, broadly speaking, the loss function may be considered to include a fifth loss function representing a JND metric, where the fifth loss function may be calculated based on the difference between the label information, including the relative perceptual difference, and the prediction.

[0077] Degradation Type Another advantage of programmatically generated data is that it is possible to know which impairments are being applied when starting from a signal that is considered clean or without significant impairments. Thus, broadly speaking, this impairment type head (often also referred to as classification head) 1126 can take the latent vectors z and further process them (e.g., via MLP 1134) to generate a probability output. It can further compute binary cross entropy using flags (e.g., label information) signaling the type of distortion in the original audio, if available. In other words, it can be considered that, broadly speaking, the loss function may include a sixth loss function representing an impairment type metric, and the sixth loss function may be calculated based on the difference between the label information including the respective impairment function information and its prediction.

[0078] More specifically, in one possible implementation, a multi-class classification loss may be constructed as follows:

number

[0079] Deterioration strength Generally speaking, this degradation intensity head 1127 (often also referred to as degradation head, as distinguishable from the classification head 1126 described above) can take the latent vectors z and further process them (e.g., by MLP 1135) to generate an output, such as a value between 1 and 5. A regression-based loss can then be calculated using the level of degradation that has been introduced into the audio, if available (e.g., from available label information). In some implementations, this level of degradation may be logged (stored) from an (automatic) degradation algorithm that has been applied prior to training of the network / system. In other words, it can be considered that, broadly speaking, the loss function may include a seventh loss function representing a degradation intensity metric, and the seventh loss function may be calculated based on the difference between the label information including the respective degradation intensity information and its prediction.

[0080] More specifically (but not by way of limitation), at the time of applying an impairment to a signal, a corresponding impairment strength can also usually be determined (and applied thereto). Thus, in a possible example, corresponding regressors may be added as follows:

number

[0081] Other quality metrics Finally, the pair {x i ,x j} has been generated, it is always possible to compute other or traditional reference-based (or reference-free) quality measures across these pairs and learn from them.

[0082] Generally speaking, this regression head 1128 can take the latent vectors z and further process them (e.g., by MLP 1136) to generate as many outputs as alternative metrics are available or pre-computed for the expected audio if available. In other words, it can be considered that, broadly speaking, the loss function may include an eighth loss function representing a regression metric, which may be calculated according to at least one of reference-based and / or non-reference-based quality measures.

[0083] In one possible implementation, pooling of regression losses may be implemented as follows:

number

number

[0084] In particular, it should be understood that each audio sample in the training set may be used in one or more (but not necessarily all) of the multiple loss functions described above. That is, some of the audio samples in the training set may be reused or shared by one or more loss functions. This is also reflected and shown in FIG. 1B. For example, the (algorithmically generated) audio sample 1142 for calculating the loss function representing the score consistency head (metric) 1123 may be reused when calculating the loss function representing the degradation condition head (metric) 1124, or vice versa. Thus, the efficiency of the system in training may be significantly improved. Furthermore, it should be noted that in some cases, the system may be further configured to generate a final (overall) loss function for the training process based on one or more of the multiple loss functions, for example by utilizing an averaging process on those loss functions. As will be appreciated by those skilled in the art, any other suitable means or processes may be used to generate such a final loss function based on any number of suitable loss functions depending on various implementations and / or requirements.

[0085] It is further noted that the multiple heads 1121-1128 described above may be constructed from either MLPs (e.g., two-layer MLPs) with any suitable number of units (e.g., 400) (perhaps all with a BN at the end) or linear layers. In some cases, it may be preferable to use a simple head to work on the encoder rather than the head, to learn high-level features that can be successfully used even by networks with limited capabilities. In some cases, the decision to use either a linear layer or an MLP may be based on the idea that the better the auxiliary work, the less capabilities the head should have. Thus, in some implementations, the linear layers for the score s (i.e., 1131) and the JND and DT heads (i.e., 1133 and 1134, respectively) may be selected empirically. Setting the linear layers for these three heads may provide properties of interest in the latent space, and the linear layers for s and L JND It reflects the "distance" between latent representations caused by L DT Note that this encourages grouping / clustering of degradation types due to the above-mentioned factors. Of course, any other suitable configuration may be applied, as will be appreciated by those skilled in the art.

[0086] FIG. 2 illustrates a deep learning algorithm for determining an audio quality metric for an audio input according to an embodiment of the present disclosure. 1 is a flow chart illustrating an example of a method 200 for training a neural network-based (e.g., neural network-based) system, which may be the same as or similar to, for example, system 100 as shown in FIG. 1A or system 110 as shown in FIG. 1B.

[0087] In particular, the method 200 begins at step S210 by obtaining as input at least one training set including audio samples. More particularly, the audio samples include a first type of audio sample and a second type of audio sample, each of the first type of audio sample being labeled with information representative of a respective predefined audio quality metric, and each of the second type of audio sample being labeled with information representative of the respective audio quality metric relative to that of a reference audio sample (e.g., that of another audio sample in the training set). As mentioned above, the reference audio sample used herein may be, but is not necessarily required to be, another audio sample in the training set. In other words, the reference audio sample may be an external reference audio sample (i.e., one not in the training set) or an internal reference audio sample (i.e., one in the training set), as will be understood and appreciated by those skilled in the art.

[0088] Such training sets containing the necessary audio samples (with appropriate label information) can be obtained (generated) in any suitable manner, as will be appreciated by those skilled in the art. For example, for the first type of audio samples, human-annotated audio data (samples, signals, frames) can be used, which can be obtained internally (e.g., by audio experts, ordinary listeners, or mechanical turkers) or externally (e.g., using publicly available data sets). As examples, such human-annotated audio data can be MOS data, JND data, etc. Further information on data sets that may be used as human-annotated can be found, for example, in Sections B.1 and B.2 of the appendix to this case. For the second type of audio samples, programmatically generated audio data (samples, signals, frames) can be used, some examples of which are described above. Further information on data sets that may be used as programmatically generated can be found, for example, in Section B.3 of the appendix to this case.

[0089] The method 200 then continues at step S220 by inputting a training set, such as input x 1000 in FIG. 1A or x 1100 in FIG. 1B, into a deep learning based (neural network based) system.

[0090] Method 200 then performs step S230 of iteratively training the system to predict label information for each of the audio samples in the training set. In particular, as described above with reference to FIG. 1B, the training may be based on multiple loss functions that may be generated to reflect differences between the label information of the audio samples in the training set and their respective predictions.

[0091] Generally speaking, the entire network / system can be trained end-to-end, for example using stochastic gradient descent and backpropagation. Prior to training, a pool of audio samples may be taken as described above and some degradation may be performed on them. As will be appreciated by those skilled in the art, various suitable degradations applied thereto may include operations / processes including, but not limited to, reverberation, clipping, encoding them using a different codec, phase distortion, reversing it, adding background noise (real or artificial). Some possible degradations are given below as examples and not by way of limitation: Additive real noise (arising from different sources) Additive artificial noise (generated colored noise) Additive Tone / Hum Noise Audio resampling ·μ-law quantization ·clipping Audio reversing -Silence Insertion Noise Insertion Damping Inserts Amplitude perturbation ·delay Equalization, bandpass and bandreject filtering Low / High Pass Filtering Chorus Overdrive Phaser Pitch Shift Reverb Tremolo Phase distortion: Griffin-Lim, random phase, phase shuffle, spectrogram holes, spectrogram convolution · Transcoding (coding with audio codec and recoding back)

[0092] In particular, the degradation can be applied in a non-stationary manner to the complete audio frame or to only a part of it. Also, possibly, some existing (automatic) measurements may be performed on these audio pairs. Although the main use of the automatically generated data is to complement the human annotated data, the disclosed network or system can still be trained without relying on one of the two and still obtain reasonable results with minimal adaptivity.

[0093] However, further information regarding possible degradation functions and optionally their corresponding degradation strengths can also be found, for example, in Section C of the appendix included herein.

[0094] The system may be trained in any suitable manner with any suitable configuration or set. For example, in some possible implementations, the system may be trained with the RangerQH optimizer, e.g., with default parameters and 10 -3 The training may be performed using a learning rate of 1 / 2. The learning rate may be decayed by a factor (e.g., 1 / 5 for 70-90% of the training). In addition, stochastic weight averaging may be employed during the last training epochs, if necessary, to promote generalization and slightly improve performance. In general, loss weighting may not be performed, since after a few iterations, all losses may be in a similar scale.

[0095] Once training is completed, the trained system may then be used or operated to determine a quality indicator metric for the input audio. Referring now to FIG. 3, a flow chart is shown illustrating an example of a method 300 for training a deep learning based (e.g., neural network based) system for determining an audio quality indicator metric for an audio input according to an embodiment of the present disclosure. The system may be the same as or similar to, for example, the system 100 as shown in FIG. 1A or the system 110 as shown in FIG. 1B. That is, the system may include a suitable encoding stage and a suitable evaluation stage as shown in either figure. The system may also go through a training process as shown in, for example, FIG. 2. Therefore, for reasons of brevity, a repetitive description thereof may be omitted.

[0096] In particular, the method 300 may begin with step S310 in which an encoding stage maps input audio samples to a feature space representation (eg, a latent space representation z as described above).

[0097] The method 300 may then continue with step 320, in which an evaluation stage predicts information representative of a given audio quality metric and information representative of a relative audio quality metric with respect to a reference audio sample based on the feature space representation. The predicted information (e.g. an indication of the audio quality metric relative to the reference audio sample) may be used to further train (regularize) the system, as described in detail above with reference to FIG. 1B.

[0098] Thus, a final quality metric such as a score (e.g., score s 1140 as shown in FIG. 1B) can be generated, such that the output metric (or score) can be used as an indication of the quality of the input audio samples. As mentioned above, the metric (or score) may be generated as any suitable representation, such as a value between 1 and 5 (e.g., either 1 or 5 indicates the best audio quality).

[0099] In summary, broadly speaking, the present disclosure proposes learning a model of speech quality that combines multiple objective variables according to a semi-supervised approach. In some cases, the disclosed approach is often simply referred to as Semi-Supervised Speech Quality Assessment (or SESQA for short). In particular, the present disclosure learns from existing labeled data, together with a (theoretically infinite) amount of unlabeled or programmatically generated data, and generates a speech quality score, together with usable latent features and useful auxiliary outputs. The score and output are simultaneously optimized in a multi-task setting by many different but complementary objective criteria, with the idea that relevant cues exist in all of them. By flowing information through a shared latent spatial bottleneck, the considered subjects learn to cooperate, promoting better and more robust representations while discarding non-essential information.

[0100] In particular, the present disclosure provides several methods, such as: As a cloud API to get the quality score of the uploaded audio -As a tool for monitoring communications - As a tool to monitor codec degradation As a (e.g. internal) tool for evaluating the performance of audio processing algorithms As a loss function for training or regularizing a deep learning model (e.g., a neural network model) As a feature extractor to know what type of distortion is present in the audio signal It is possible to use (but is not limited to) the following: Of course, any other suitable use case may be utilized, as will be understood and appreciated by those of ordinary skill in the art.

[0101] 4-8 are examples showing various results and comparisons according to embodiments of the present disclosure, respectively. In particular, quantitative comparisons are made using a number of existing or conventional approaches. In particular, details regarding some of the existing approaches used for the comparisons can be found, for example, in Section D of the appendix included herein.

[0102] It should be noted that for the evaluation, the present disclosure generally uses three MOS data sets, two internal and one publicly available. The first internal data set consists of 1,109 recordings and a total of 1.5 hours of audio, characterizing primarily user-generated content (UGC). The second internal data set consists of 8,016 recordings and 15 hours of audio, characterizing telephone and VoIP impairments. The third data set is TCD-VoIP, which consists of 384 recordings and 0.7 hours of audio, characterizing multiple VoIP impairments. Another data set we use is the JND data set, which consists of 20,797 pairs of recordings and 28 hours of audio. Further details of the training set can be found, for example, in Section B of the appendix included herewith. For programmatic data generation, the present disclosure generally uses a pool of internal and public data sets to generate 70,000 quadruples that fit 78 hours of audio. Furthermore, a total of 37 possible impairments are used, including additive background noise, hum noise, clipping, sound effects, packet loss, phase distortion, and multiple audio codecs (further details can be found, for example, in Section C of the appendix included with this application). The present disclosure is then compared to ITU-P563, using two approaches based on feature loss, one using JND (FL-JND) and the other using PASE (FL-PASE), SRMR, Auto-MOS, Quality-Net, WEnets, CNN-ELM, and NISQA. For the evaluation, some of them have been reimplemented to fit the training and evaluation pipeline of the present disclosure and adapted to operate at 48 kHz, where necessary / possible. It should be noted that FL, AutoMOS, and NISQA generally utilize partial additional data other than MOS and are therefore weakly semi-supervised approaches. Further details on the baseline approaches can be found, for example, in Section D of the appendix included with this application.

[0103] All approaches are placed under the same settings, choosing the best optimizer and hyper-parameters on the validation set. Data augmentation is performed and trained on 1 second weakly labeled frames for 5 epochs by reusing MOS data within an epoch (e.g., an epoch may be defined to go completely through the programmatically generated data). Random scaling, phase reversal, and temporal sampling may also be used as data augmentation. Evaluation involves L MOS and L CONS is used, and the proportion of incorrectly classified rankings, R RANK is calculated (for the sake of illustration, L RANK Instead of R RANK (The total error is reported.) Furthermore, we E TOTAL =0.5L MOS +R RANK +L CONS (A weight of 0.5 is introduced to compensate for the different ranges). Five-fold cross-validation is also performed and the average error is reported.

[0104] Of course, it should be understood that any other suitable training data sets and / or evaluation means may be employed according to various implementations and / or requirements.

[0105] According to these results, the approach disclosed in this work appears to outperform the evaluation metrics considered. It is also observed that the scores obtained from the score-head correlate well with human judgements of quality, that they are able to detect different levels of degradation for many distortions, and that the latent space z clusters degradation types.

[0106] For example, Figure 4 shows that in general, the scores appear to correlate well with human judgment.

[0107] Figure 5 shows the empirical distribution of distances between latent space vectors z. It can be seen from diagram 510 that smaller distances correspond to similar utterances with the same type and intensity of impairment (e.g., mean distance 7.6 and standard deviation 3.4), and from diagram 530 that larger distances correspond to different utterances with different impairments (e.g., mean distance 16.9 and standard deviation 3.9). The overlap between the two is small, and the mean plus one standard deviation does not intermingle with each other. Similar utterances with different impairments (diagram 520) are spread between the two aforementioned distributions (e.g., mean distance 13.7 and standard deviation 5.5). This makes sense in a latent space organized by impairment and intensity, with a wide range between small and large intensities. It can be hypothesized that this overall behavior may be the result of all the losses, especially s and LJND and their (linear) heads.

[0108] Figure 6A shows how the score s calculated from the test signal without impairments tends to decrease as the impairment strength increases. In many cases, the impact seems clear and consistent (e.g. additive noise or EAC3 codec). In other cases, for high strengths (e.g. μ-law quantization or clipping), the effect seems to saturate. There also seem to be a few impairments whose strength does not correspond to a single variable, and their impact seems less clear. Overall, a consistent behavior between impairments and strength is observed. L MOS ,L RANK ,L DS It can be hypothesized that may be the main driving force for achieving this behavior. Figures 6B and 6C show schematics of similar additional results, where the scores seem to adequately reflect the progressive audio degradation.

[0109] Figure 7A shows three low-dimensional t-SNE projections of the latent space vector z. In this figure, we can see how the different impairment types are grouped or clustered together. For example, we can see that for a perplexity of 200, the latent vectors of frames with additive noise are grouped together in the center. Interestingly, we can also see that similar impairments can be located close to each other. This is the case for example for additive and colored noise, MP3 and OPUS codecs, or Griffin-Lim and STFT phase distortions, respectively. This clustering behavior is consistent with the L DT It can be hypothesized that this may be a direct result of its (linear) head.

[0110] FIG. 7B shows a schematic of similar additional results where the classification head may distinguish between types of degradation.

[0111] FIG. 8A shows a schematic comparison with several existing or conventional approaches. From FIG. 8A, it can be observed that, overall, all approaches appear to clearly outperform the random baseline, and that about half of them appear to achieve an error comparable to the variability between human scores (L MOS (The L is estimated by taking the standard deviation across listeners and averaging it over the entire utterance.) Many existing approaches also CONS We also observe that existing approaches report reasonable consistency for R RANK This disclosure introduces significant error when considering standard L MOS It appears to outperform all existing approaches listed by a large margin in all the metrics considered, including the L. The only exception to the above statement is the L of the ITU-P563 approach. CONS It seems to be about metrics, but still, this is a high L CONSand almost random R RANK Summary Metric E TOTAL Taking this into account, the present disclosure appears to cut the error of the best existing approach by 36%.

[0112] FIG. 8B shows a schematic of the impact of the considered criteria / tasks on the performance of the disclosed method of the present disclosure. First, it is observed that by removing one criterion, the error never seems to decrease. This may indicate that none of them seem to be detrimental from a performance point of view. Second, there are some related criteria (e.g., L MOS and L RANK ) has been observed. However, the absence of one such relevant criterion has not yet led to the mean error of existing approaches (see, e.g., Fig. 8A). Note that for some less relevant tasks, they still seem to be useful, either for the outputs they generate or for the properties they impart to the organization of the latent space z (e.g., knowing whether a pair of signals resulted in a JND difference). Finally, L MOS It is also interesting to highlight that when only the criteria are considered (see the last row of FIG. 8B), performance appears to be comparable to some existing best-performing approaches (see, for example, NISQA and CNN-ELM in FIG. 8A). Overall, this demonstrates that considering multiple optimization criteria and tasks seems to be the key to achieving good performance, experimentally justifying a semi-supervised approach to audio quality assessment such as the present disclosure.

[0113] FIG. 8C shows in a schematic way the results of further evaluating the generalization ability of the envisaged approach by performing a post-hoc informal test with out-of-sample data. For that purpose, for example, 20 new recordings from UGC may be selected, which characterize clean or production quality speech and speech with impairments such as real background noise, codec artifacts, or microphone distortion. The new set of listeners can then be asked to rate the quality of the recordings with a score between 1 and 5 and compare their ratings with those predicted by a model pre-trained on our internal UGC data set. It can be seen from FIG. 8C that the ranking of existing approaches changes, with some showing better generalization to out-of-sample data than others. Nevertheless, the present disclosure still seems to outperform them by a large margin in all enumerated metrics. In particular, the L of the best enumerated existing approaches is significantly better than the L of the best enumerated ones. MOS appears to cut 21%, and the best existing value is the relative L observed for the in-sample data. MOS This is significantly larger than the difference, which was 7% (from Figure 8A), indicating that our method generalizes better to out-of-sample but relevant data.

[0114] Figures 8D and 8E further show the error values ​​for a hypothetical data set, plotted against the L across the data set. TOTAL In particular, FIG. 8D shows a schematic comparison of the present disclosure with existing approaches, and FIG. MOS In addition to using only the sigma-based loss, we also show diagrammatically the effect of training without one of the assumed losses. In particular, similar to the one mentioned above, E TOTAL =0.5L MOS +R RANK +L CONS 8F further provides some additional results that show, in general terms, that the proposed approach of the present disclosure (last row) appears to be superior to the conventional approaches listed.

[0115] Above, possible methods of training and operating a deep learning-based (e.g., neural network-based) system for determining an audio quality measure of an input audio sample, as well as possible implementations of such a system, are described. Furthermore, the present disclosure also relates to apparatus for performing these methods. Examples of such apparatus may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application specific integrated circuits (ASICs), one or more radio frequency integrated circuits (RFICs), or any combination thereof) and a memory coupled to the processor. The processor may be configured to perform all or a portion of the steps of the methods described throughout the present disclosure.

[0116] An apparatus may be a server computer, a client computer, a personal computer (PC), a tablet PC, a set top box (STB), a personal digital assistant (PDA), a cellular phone, a smartphone, a web appliance, a network router, a switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by the apparatus. Furthermore, the present disclosure is intended to relate to any collection of apparatuses that individually or together execute instructions to perform any one or more of the methods described herein.

[0117] The present disclosure further relates to a program (e.g., a computer program product) comprising instructions that, when executed by a processor, cause the processor to perform all or a portion of the steps of the methods described herein.

[0118] Additionally, the present disclosure relates to a computer-readable (or machine-readable) storage medium that stores the above-mentioned program, where the term "computer-readable storage medium" includes, but is not limited to, data repositories in the form of solid-state memories, optical media, and magnetic media.

[0119] Unless specifically stated otherwise, as will be apparent from the discussion that follows, use of terms such as "processing," "computing," "calculating," "determining," "analyzing," and the like throughout the description of this disclosure will be recognized to refer to operations and / or processes of a computer or computing system or similar electronic computing device that manipulates and / or transforms data expressed as physical quantities (e.g., as electrons) into other data similarly expressed as physical quantities.

[0120] Similarly, the term "processor" may refer to any device or part of a device that processes electronic data, e.g., to convert electronic data from registers and / or memory into other electronic data that can be stored in registers and / or memory. A "computer" or "computing machine" or "computing platform" may include one or more processors.

[0121] The methods described herein may, in an exemplary embodiment, be executed by one or more processors that accept computer-readable (also referred to as machine-readable) code that includes a set of instructions, which, when executed by one or more processors, perform at least one of the methods described herein. Any processor capable of executing a series of instructions (sequential or otherwise) that specify operations to be performed is included. Thus, one example is a typical processing system that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem that includes a main RAM and / or static RAM, and / or a ROM. A bus subsystem may be included for communicating between components. The processing system may also be a distributed processing system with processors coupled by a network. If the processing system requires a display, a display such as, for example, a liquid crystal display (LCD) or a cathode ray tube (CRT) display may be included. If manual data entry is required, the processing system may also include an input device such as one or more of an alphanumeric input unit such as a keyboard, a pointing control device such as a mouse, and the like. The processing system may also include a storage system, such as a disk drive unit. The processing system in some configurations may include an audio output device and a network interface device. Thus, the memory subsystem includes a computer-readable carrier medium that carries computer-readable code (e.g., software) that includes a set of instructions that, when executed by one or more processors, cause the processor to perform one or more of the methods described herein. It should be noted that when a method includes several elements, e.g., several steps, no order of such elements is implied unless specifically stated.The software may reside in a hard disk, or may reside completely or at least partially in the RAM and / or in the processor during execution by the computer system. Thus, the memory and the processor also constitute a computer-readable carrier medium carrying computer-readable code. Furthermore, the computer-readable carrier medium may form or be included in a computer program product.

[0122] In alternative exemplary embodiments, one or more processors may operate as stand-alone devices or may be connected in a networked deployment, e.g., networked to other processors, one or more processors may operate in the capacity of a server or user machine in a server-user network environment, or as a peer machine in a peer-to-peer or distributed network environment. One or more processors may be a personal computer (PC), a tablet PC, a personal digital assistant (PDA), a cellular phone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be performed by the machine.

[0123] It should be noted that the term "machine or machine" is also intended to include any collection of machines that individually or collectively execute a set (or sets) of instructions for performing any one or more of the methods described herein.

[0124] Thus, an exemplary embodiment of each method described herein is in the form of a computer readable carrier medium carrying a set of instructions, e.g., a computer program, for execution on one or more processors, e.g., one or more processors that are part of a web server configuration. Thus, as will be appreciated by those skilled in the art, exemplary embodiments of the present disclosure can be embodied as a method, an apparatus, such as a special purpose device, an apparatus, such as a data processing system, or a computer readable carrier medium, e.g., a computer program product. The computer readable carrier medium carries computer readable code, which, when executed on one or more processors, causes the processor or processors to perform the method.

[0125] Accordingly, aspects of the present disclosure may take the form of a method, an entirely hardware exemplary embodiment, an entirely software exemplary embodiment, or an exemplary embodiment combining software and hardware aspects. Further, the present disclosure may take the form of a carrier medium carrying computer readable program code embodied in the medium (e.g., a computer program product in a computer readable storage medium).

[0126] The software may also be transmitted and received over a network via a network interface device. While the carrier medium is a single medium in an exemplary embodiment, the term "carrier medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store one or more sets of instructions. The term "carrier medium" should also be taken to include any medium capable of storing, encoding, or carrying a set of instructions for execution by one or more processors, causing one or more processors to perform any one or more of the methods of the present disclosure. Carrier media can take many forms, including, but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical disks, magnetic disks, and magneto-optical disks. Volatile media include dynamic memory, such as main memory. Transmission media include coaxial cables, copper wire, and fiber optics, including the wires that make up a bus subsystem. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications. Thus, for example, the term "carrier medium" shall be taken to include, but is not limited to, computer products embodied in solid state memories, optical and magnetic media; media carrying a propagated signal detectable by at least one processor or by one or more processors and representing a set of instructions that, when executed, implements a method; and transmission media within a network carrying a propagated signal detectable by at least one processor of one or more processors and representing a set of instructions.

[0127] It will be appreciated that the steps of the described method are performed, in an exemplary embodiment, by a suitable processor (or processors) of a processing system (e.g., a computer) executing instructions (computer-readable code) stored in storage. It will also be appreciated that the present disclosure is not limited to any particular implementation or programming technique, and that the present disclosure may be implemented using any suitable technique for implementing the functions described herein. The present disclosure is not limited to any particular programming language or operating system.

[0128] Throughout this disclosure, references to "one exemplary embodiment," "some exemplary embodiments," or "exemplary embodiments" mean that a particular feature, structure, or characteristic described in connection with an exemplary embodiment is included in at least one exemplary embodiment of the disclosure. Thus, the appearances of the phrases "in one exemplary embodiment," "some exemplary embodiments," or "exemplary embodiments" in various places throughout this disclosure are not necessarily all referring to the same exemplary embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more exemplary embodiments, as would be apparent to one of ordinary skill in the art from this disclosure.

[0129] As used herein, unless otherwise specified, the use of ordinal adjectives "first," "second," "third," etc. to describe a common object merely indicates that different instances of similar objects are being referred to and is not intended to imply that the objects so described must be in a given order in time or space, in a ranking, or in any other manner.

[0130] In the following claims and the description of this case, any one of the terms including, including, or including is an open term meaning to include at least one element / feature that follows without excluding others. Therefore, when used in the claims, the term including should not be interpreted as being limited to the means, elements, or steps listed thereafter. For example, the scope of the expression device including A and B should not be limited to a device consisting of only elements A and B. As used in this case, any one of the terms including, thing including, or something that includes is also an open term meaning to include at least one element / feature that follows without excluding others. Therefore, including is synonymous with and means having.

[0131] In the above description of exemplary embodiments of the present disclosure, it should be appreciated that various features of the disclosure are often grouped together in a single exemplary embodiment, figure, or description thereof for the purpose of facilitating the flow of the disclosure and aiding in the understanding of one or more various inventive aspects. However, this method of disclosure should not be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single disclosed exemplary embodiment as set forth above. Accordingly, the claims following the specification are hereby expressly incorporated into this specification, with each claim standing on its own as an individual exemplary embodiment of the present disclosure.

[0132] Furthermore, although some exemplary embodiments described herein include some features that are not included in other exemplary embodiments, as will be understood by those skilled in the art, combinations of features of different exemplary embodiments are intended to fall within the scope of the present disclosure and form different exemplary embodiments. For example, in the following claims, any claimed exemplary embodiment can be used in any combination.

[0133] In the description provided herein, numerous specific details are set forth. However, it will be understood that exemplary embodiments of the present disclosure may be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of the present description.

[0134] Thus, while what is believed to be the best mode of the disclosure has been described, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of the disclosure, and all such variations and modifications are intended to be claimed as falling within the scope of the disclosure. For example, any formulas shown above are merely representations of procedures that may be used. Functionality may be added or removed from the block diagrams, and operations may be interchanged between functional blocks. Steps may be added or removed to methods described within the scope of the disclosure.

[0135] Enumerated exemplary embodiments of the present disclosure ("EEE") are described above in relation to a method and system for determining an audio quality measure of an audio input. Accordingly, embodiments of the present invention may relate to one or more of the following enumerated examples:

[0136] EEE1. 1. A method for training a convolutional neural network (CNN) for determining an audio quality assessment of an audio signal, the method comprising: Transforming the audio signal into a low-dimensional latent space representation of the audio signal; inputting a low-dimensional latent space representation of the audio signal into an encoder stage; processing the low dimensional latent space representation of the audio signal by an encoder stage to determine parameters of the low dimensional latent space representation of the audio signal; determining an audio quality score for the audio signal based on the parameters and the low-dimensional latent space representation of the audio signal; The method includes:

[0137] EEE2. 1. A method of training a deep learning based system to determine a metric of audio quality for an audio input, the method comprising: obtaining as input at least one training set comprising audio samples, the audio samples comprising a first type of audio samples and a second type of audio samples, each of the first type of audio samples being labeled with information representative of a respective predefined audio quality metric, and each of the second type of audio samples being labeled with information representative of the respective audio quality metric relative to that of a reference audio sample or relative to that of another audio sample in the training set; inputting the training set into a deep learning based system; and iteratively training the system to predict label information for each of the audio samples in the training set; where the training is based on multiple loss functions; and In this method, multiple loss functions are generated to reflect the difference between the label information of the audio samples in the training set and their respective predictions.

[0138] EEE3. In the IEEE 2 method, a first type of audio samples includes human-annotated audio samples, each of which is labeled with information representative of a respective predefined audio quality metric.

[0139] EEE4. In the IEEE 3 method, the human-annotated audio samples include Mean Opinion Score (MOS) audio samples and / or Just Noticeable Difference (JDN) audio samples.

[0140] EEE5. In the method according to any one of the IEEE standards above, the second type of audio samples includes algorithmically generated audio samples, each of which is labeled with information representative of a relative predetermined audio quality metric.

[0141] EEE6. In the method according to IEEE 5, each of the algorithmically generated audio samples is generated by selectively applying at least one degradation function, each with a respective degradation strength, to a reference audio sample or to an audio sample generated by another algorithm, and the label information includes information indicating each degradation function that has been applied and / or the respective degradation strength.

[0142] EEE7. In the method according to IEEE 6, the label information further comprises information representing the degradation relative to other audio samples in the training set or relative to a reference audio sample.

[0143] EEE8. In the IEEE 6 or 7 methods, the degradation function is selected from among a number of available degradation functions and / or the respective degradation strength is set such that at its minimum value the degradation is perceptually noticeable.

[0144] EEE9. In the IEEE 8 method, the multiple available degradation functions include functions related to one or more of reverberation, clipping, encoding using different codecs, phase distortion, audio reversal, and background noise.

[0145] EEE10. In the method according to any one of IEEE 6 to 9, the audio frames generated by the algorithm are arranged in pairs of audio frames {x i ,x j} and / or quadruple audio frames {x ik ,x il ,x jk ,x jl} and audio frame x i is generated by selectively applying at least one degradation function, each with a respective degradation strength, to a reference audio frame, j applies at least one degradation function, each with a respective degradation strength, to an audio frame x i The audio frame x is generated by selectively applying ik ,x il Let x be the time delay for each audio frame. i By selectively applying i Extracted audio frame x jk ,x jl is the time delay for each audio frame x j By selectively applying j is extracted from.

[0146] EEE11. In the method according to any one of the above IEEE paragraphs, the loss function includes a first loss function representing a MOS error metric, the first loss function being calculated based on a difference between a MOS ground truth for the audio samples in the training set and a prediction of the audio samples.

[0147] EEE12. In the method according to any one of IEEE Clauses 5 to 10, or IEEE 11 when dependent on any one of IEEE Clauses 5 to 10, the label information of the second type of audio samples includes relative information indicating whether one audio sample is more degraded than another audio sample, and the loss function includes a second loss function representing a pairwise ranking metric, the second loss function being calculated based on a ranking established by the label information including the relative degradation information and a prediction thereof.

[0148] EEE13. In the IEEE 12 method, the system is trained in such a way that one audio sample that is less degraded obtains an audio quality metric that represents better audio quality than another audio sample that is more degraded.

[0149] EEE14. In the method according to any one of IEEE Clauses 5 to 10, 12 and 13, or IEEE 11 when dependent on any one of IEEE Clauses 5 to 10, the label information of the second type of audio samples includes relative information representing perceptual relevance between the audio samples, and the loss function includes a third loss function representing a consistency metric, and the third loss function is calculated based on a difference between the label information including the perceptual relevance information and the prediction.

[0150] EEE15. In the IEEE 14 method, the consistency metric indicates whether two or more audio samples have the same degradation function and degradation strength and correspond to the same time frame.

[0151] EEE16. In the method according to any one of IEEE Clauses 5 to 10 and 12 to 15, or IEEE 10 when dependent on any one of IEEE Clauses 5 to 10, the label information of the second type audio sample includes relative information representing whether an audio sample is applied with the same degradation function and the same degradation strength as another audio sample, and the loss function includes a fourth loss function representing a degradation state metric, and the fourth loss function is calculated based on a difference between the label information including the relative degradation information and the prediction.

[0152] EEE17. In the method according to any one of IEEE Clauses 5 to 10 and 12 to 16, or IEEE 11 when dependent on any one of IEEE Clauses 5 to 10, the label information of the second type audio samples includes relative information representing perceptual differences with respect to each other, and the loss function includes a fifth loss function representing a JND metric, the fifth loss function being calculated based on a difference between the label information including the relative perceptual differences and the prediction.

[0153] EEE18. The method according to any one of IEEE clauses 5 to 10 and 12 to 17, or IEEE 11 when dependent on any one of IEEE clauses 5 to 10, wherein the label information of the second type audio samples includes information representative of a degradation function applied to the audio samples, and the loss function includes a sixth loss function representative of a degradation type metric, and the sixth loss function is calculated based on a difference between the label information including the information of the respective degradation function and its prediction.

[0154] EEE19. In the method according to any one of IEEE clauses 5 to 10 and 12 to 18, or IEEE 11 when dependent on any one of IEEE clauses 5 to 10, the label information of the second type audio samples includes information representative of a degradation intensity applied to the audio samples, and the loss function includes a seventh loss function representative of a degradation intensity metric, and the seventh loss function is calculated based on a difference between the label information including the respective degradation intensity information and the prediction.

[0155] EEE20. In the method according to any one of the above IEEE paragraphs, the loss function includes an eighth loss function representing a regression metric, the regression metric being calculated according to at least one of a reference-based quality measure and a non-reference based quality measure.

[0156] EEE21. In the IEEE 20 method, the reference-based quality measures include at least one of PESQ, CSIG, CBAK, COVL, SSNR, LLR, WSSD, STOI, SISDR, Mel-Cepstral Distortion, and Log-Mel-Band Distortion.

[0157] EEE22. In the method according to any one of the above IEEE paragraphs, each of the audio samples in the training set is used in at least one of the multiple loss functions, and a final loss function for training is generated based on an averaging process for one or more of the multiple loss functions.

[0158] EEE23. In a method according to any one of the above IEEE standards, the system includes an encoding stage for mapping an audio input into a feature space representation, and an evaluation stage for generating a prediction of label information based on the feature space representation.

[0159] EEE24. The method according to any one of the above IEEE claims, wherein the encoding stage for generating the intermediate representation comprises a neural network encoder.

[0160] EEE25. In the method according to any one of the above IEEE paragraphs, each of the plurality of loss functions is determined based on a neural network including a linear layer or a multi-layer perceptron (MLP).

[0161] EEE26. 1. A deep learning based system for determining an audio quality metric for input audio samples, the system comprising: An encoding stage; and Evaluation stage; the encoding stage being configured to map input audio samples to a feature space representation; and The evaluation stage is configured to predict information representative of a predetermined audio quality metric based on the feature space representation, and to further predict information representative of a relative audio quality metric for another audio sample, the system.

[0162] EEE27. In a system according to IEEE 26, the system: obtaining as input at least one training set, the training set including audio samples of a first type and audio samples of a second type, each of the audio samples of the first type being labeled with information representative of a respective predefined audio quality metric, and each of the audio samples of the second type being labeled with information representative of the respective audio quality metric relative to that of a reference audio sample or to that of another audio sample in the training set; inputting a training set into the system; and iteratively training the system based on the training set to predict label information for each of the audio samples in the training set based on a plurality of loss functions, the plurality of loss functions being generated to reflect differences between the label information of the audio samples in the training set and their respective predictions; A system that is configured to run

[0163] EEE28. 1. A method of operating a deep learning based system for determining an audio quality measure for input audio samples, the system including an encoding stage and an evaluation stage, the method comprising: mapping the input audio samples to a feature space representation by an encoding stage; and predicting, by an evaluation stage, information representative of the given audio quality metric and information representative of a relative audio quality metric with respect to another audio sample based on the feature space representation; The method includes:

[0164] EEE29. A program including instructions, which when executed by a processor, cause the processor to perform steps of a method according to any one of IEEE 1-15 and 28.

[0165] EEE30. A computer-readable storage medium storing a program conforming to IEEE 29.

[0166] EEE31. An apparatus comprising a processor and a memory coupled to the processor, the processor configured to cause the apparatus to perform the steps of a method according to any one of IEEE 1-25 and 28.

[0167] appendix Appendix A: Score calculation using reference signals To compute the score s in a reference-based setting, rather than a reference-free one, we use two signals x i ,x j is passed through the encoder to generate the corresponding latent point z i ,z j Then, for example, we use a linear unit on both latent points to obtain

number

number

number

[0168] Appendix B: Data As already mentioned, three types of data are used in the semi-supervised approach: MOS data, JND data, and programmatically generated data. An additional out-of-sample data set used in the post-hoc listening tests is summarized in the description, and its degradation characteristics are similar to those in the internal UGC data set (see below).

[0169] B.1. MOS Data The entire network / system is trained and evaluated on three different MOS data sets of different sizes and features: 1. Internal UGC Data Set - This data set consists of 1,109 recordings of UGC, adding up to a total of 1.5 hours of audio. All recordings are converted to mono WAV PCM at 48 kHz and normalized to have the same loudness. The utterances range from single words to several lines, are spoken by both male and female speakers under various conditions, and use different languages ​​(mostly English, but sometimes Chinese, Russian, Spanish, etc.). Common impairments in the recordings include background noise (street, cafeteria, wind, background TV / radio, other people talking, etc.), reverberation, bandwidth reduction (low pass drop up to 3 kHz), and coding artifacts (MP3, OGG, AAC, etc.). Quality assessments were collected with the help of a group of 10 expert listeners with at least several years of experience in audio processing / engineering. The recordings ranged between 4 and 10 ratings and were obtained by following standard procedures similar to those described by the IEEE and ITU (see the following paper and its references): PC Loizou, “Speech quality assessment,” in Multimedia Analysis, Processing and Communications, ser. Studies in Computational Intelligence. Berlin, Germany: Springer, 2011, vol. 346, pp. 623-654

[0170] 2. Internal Telephone / VoIP Data Set - This data set consists of 8,016 recordings with typical telephone and VoIP impairments, adding up to a total of 15 hours of audio. All audio, except for a small percentage, was initially recorded at 48 kHz, then processed and normalized to have the same loudness. Recordings contain two sentences separated by silence, have durations between 5 and 15 seconds, and follow a protocol similar to ITU-P800. Male and female speech is balanced, and a variety of languages ​​are present (English, French, Italian, Czech, etc.). Common impairments include packet loss (20-60 ms), bandwidth reduction (low-pass drop to 3 kHz), additive synthetic noise (different SNRs), and coding artifacts (G772, OPUS, AC3, etc.). Quality ratings are provided by a panel of regular listeners, with each recording receiving a rating between 10 and 15. The evaluation was obtained by following the standard procedures described by the ITU (see the following document and its references): PC Loizou, “Speech quality assessment,” in Multimedia Analysis, Processing and Communications, ser. Studies in Computational Intelligence. Berlin, Germany: Springer, 2011, vol. 346, pp. 623-654.

[0171] 3.TCD-VoIP Data Set - This is http: / / www:mee:tcd:ie / ~ sigmedia / Resources / TCD-VoIP The data set is publicly available online at http: / / www.voip.org / . It consists of 384 recordings with typical VoIP impairments totaling up to 0.7 hours. A good description of the data set is given in the original reference. (N. Harte, E. Gillen, and A. Hines, “TCD-VoIP, a research database of degraded speech for assessing quality in VoIP applications,” in Proc. of the Int. Workshop on Quality of Multimedia Experience (QoMEX), 2015) Although the VoIP impairments are distinct, many of them differ (both in type and intensity) from our internal telephone / VoIP data set.

[0172] B.2. JND Data JND data is also used for training. We used the dataset compiled by Manocha et al. (P. Manocha, A. Finkelstein, Z. Jin, NJ Bryan, R. Zhang, and GJ Mysore, “A differentiable perceptual audio metric learned from just noticeable differences,” ArXiv:2001.04460, 2020), which is available at: https: / / github:com / pranaymanocha / PerceptualAudio The data set consists of 20,797 pairs of “perturbed” recordings (28 hours of audio), each from the same utterance, accompanied by annotations of whether such perturbations were noticeable for each pair. The annotations were crowdsourced from Amazon Mechanical Turk, which follows a specific procedure (P. Manocha, A. Finkelstein, Z. Jin, NJ Bryan, R. Zhang, and GJ Mysore, “A differentiable perceptual audio metric learned from just noticeable differences,” ArXiv:2001.04460, 2020). The perturbations correspond to additive linear background noise, reverberation, and coding / compression.

[0173] B.3. Programmatically Generated Data Quadruple {x ik ,x il ,x jk ,x jl} is calculated from the data generated by the program. For that, a list of 10 data sets of audio at 48 kHz is used, which are considered clean and unprocessed. This includes private / proprietary and public data sets, the latter being e.g.: VCTK (Y. Yamagishi, C. Veaux, and K. MacDonald, “CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice cloning toolkit (version 0.92),” University of Edinburgh, The Center for Speech and Technology Research (CSTR), 2019. [Online]. Available: https: / / doi:org / 10:7488 / ds / 2645), RAVDESS (SR Livingstone and FA Russo, “The Ryerson audio-visual database of emotional speech and song (RAVDESS),” PLoS ONE, vol. 13, no. 5, p. e0196391, 2018. [Online]. Available: https: / / zenodo:org / record / 1188976), or TSP Speech (http: / / www-mmsp:ece:mcgill:ca / Documents / Data / ). For the experiments of this disclosure, 50,000 quadruples are used for training, 10,000 for validation, and 10,000 for testing. To generate all quadruples, one may proceed as follows:

[0174] Uniformly sample a data set and then uniformly sample a file from it. Uniformly sample a 1.1 second frame, avoiding silent or mostly silent frames. Normalize it to have a maximum absolute amplitude of 1. With probabilities 0.84, 0.12, and 0.04, sample an impairment of 0, 1, or 2 from the pool of available impairments (see below). In the case of zero impairment, the signal immediately changes to x i Otherwise, we choose the strengths uniformly for each degradation and set x i Apply them in order to produce With probabilities 0.75, 0.2, 0.04, 0.01, sample 1, 2, 3, or 4 impairments from the pool of available impairments (see below). Select the intensities uniformly and multiply them by x i Apply to x in order j Generate. · Sample the time delay uniformly between 0 and 100 ms. With such a delay, x i Frame x from 1s ik ,x ilExtract and do the same with x i From x jk ,x jl This will be done. ·{x ik ,x il ,x jk ,x jl} along with information on the degradation type and intensity. Total of 78 hours of audio: 1 x 4 x (50000+10000+10000) / 3600 = 77:77 h used:

[0175] Appendix C: Aging and Strength 37 possible impairments were considered along with their corresponding intensities. The intensities were set by the author at the minimum value so that they were perceptually noticeable. Note that in some cases, the intensities selected below are only one aspect of the overall impairments, and for other relevant aspects we randomly sample between empirically chosen values. For example, in the case of the reverberation effect, the SNR is selected as the main strength, but also the reverberation type, width, delay, etc. are randomly selected.

[0176] 1. Additive noise - With probability 0.29, sample a noise frame from the available pool of noise data sets and add it to x with SNR between 35 and -15 dB. The noise data sets include private / proprietary and public data sets, the latter being e.g.: ESC (KJ Piczak, “ESC: dataset for environmental sound classification,” in Proc. of the ACM Conf. on Multimedia (ACM-MM), 2015, pp. 1015-1018. [Online]. Available: https: / / doi:org / 10:7910 / DVN / YDEPUT) or FSDNoisy18k (E. Fonseca, M. Plakal, DPWE Ellis, F. Font, X. Favory, and X. Serra, “Learning sound event classifiers from web audio with noisy labels,” ArXiv: 1901.01189, 2019. [Online]. Available: https: / / doi:org / 10:5281 / zenodo:2529934). This impairment can be applied to the entire frame, or to only a portion of the frame (minimum 300 ms) with probability 0.25.

[0177] 2. Colored noise - Generate a colored noise frame with a uniform exponent between 0 and 0.7 with probability 0.07. Add it to x with an SNR between 45 and -15 dB. This impairment can be applied to the whole frame or only to a part of the frame (minimum 300 ms) with probability 0.25.

[0178] 3. Hum Noise - Adds a tone (sine, sawtooth, square) around 50 or 60 Hz with a probability of 0.035 and an SNR between 35 and -15 dB. This impairment can be applied to the whole frame or only to a portion of the frame (minimum 300 ms) with a probability of 0.25.

[0179] 4. Tonal Noise - Probability 0.011, does the same as above, but with frequencies between 20 and 12,000 Hz.

[0180] 5.Resampling - With probability 0.011, resample the signal to a frequency between 2 and 32 kHz and convert it to 48 kHz.

[0181] 6. μ-law quantization - applies μ-law quantization between 2 and 10 bits with probability 0.011.

[0182] 7. Clipping - Probability 0.011, clips between 0.5 and 99% of the signal.

[0183] 8. Audio Reverse – Reverses the signal in time with probability 0.05.

[0184] 9. Silence Insertion - Inserts between 1 and 10 silent sections of length between 20 and 120 ms with probability 0.011.

[0185] 10. Noise injection - Same as above but with white noise, with probability 0.011.

[0186] 11. Damping Insert - with probability 0.011, do the same as above but dampen the section by multiplying it by a maximum linear gain of 0.8.

[0187] 12. Amplitude perturbation - with probability 0.011, do the same as above but inject multiplicative Gaussian noise.

[0188] 13. Sample Replication - with probability 0.011, do the same as above but repeat the previous difference pull.

[0189] 14. Delay – Adds delayed versions of the signal (single tap and multi tap) using a maximum delay of 500ms with probability 0.035.

[0190] 15. Extreme Equalization - applies an equalization filter with random Q and gain greater than 20 dB or less than -20 dB with probability 0.006.

[0191] 16. Band Pass - Applies a random Q band pass filter at a random frequency between 100 and 4000 Hz with probability 0.006.

[0192] 17. Band Reject - with probability 0.006, does the same as above but rejects the band.

[0193] 18. High Pass – Applies a random Q band pass filter at a random frequency between 150 and 4000 Hz with probability 0.011.

[0194] 19. Low Pass – Applies a low pass filter with a random cutoff frequency between 250 and 8000 Hz with probability 0.011.

[0195] 20. Chorus - Adds a chorus effect with a linear gain between 0.15 and 1 with a probability of 0.011.

[0196] 21. Overdrive - Adds an overdrive effect with a gain between 12 and 50 dB, with a probability of 0.011.

[0197] 22.Phasor - Adds a phaser effect with a probability of 0.011 and a linear gain between 0.1 and 1.

[0198] 23. Reverberation - Adds reverberation with a probability of 0.035 and with a SNR between -5 and 10 dB.

[0199] 24. Tremolo - Adds a tremolo effect with a depth between 30 and 100% with a probability of 0.011.

[0200] 25. Griffin-Lim Reconstruction - Perform a Griffin-Lim reconstruction of the STFT of the signal with probability 0.023. The STFT is computed using a random window length and 50% overlap.

[0201] 26. Phase Randomization - With probability 0.011, does the same as above but uses random phase information.

[0202] 27. Phase Shuffle - With probability 0.011, do the same as above, but shuffle the window phase in time.

[0203] 28. Spectrogram Convolution - Convolve the STFT of the signal with a 2D kernel with probability 0.011. The STFT is computed using a random window length and 50% overlap.

[0204] 29. Spectrogram Hole - Applies dropout to spectral amplitudes between 0.15 and 0.98 with probability 0.011.

[0205] 30. Spectrogram Noise - With probability 0.011, does the same as above, but replaces zeros with random values.

[0206] 31. Transcoding MP3 - Encode to MP3 using libmp3lame with probability 0.023 and back using between 2 and 96 kbps (all codecs come from ffmpeg).

[0207] 32. Transcoding AC3 - With probability 0.035, encode to AC3 and back using between 2 and 96 kbps.

[0208] 33. Transcoding EAC3 - with probability 0.023, encode to EAC3 and back using between 16 and 96 kbps.

[0209] 34. Transcode MP2 - With probability 0.023, encode to MP2 and back using between 32 and 96 kbps.

[0210] 35. Transcode WMA - with probability 0.023, encode to WMA and back using between 32 and 128 kbps.

[0211] 36. Transcoding OGG - With probability 0.023, encode to OGG and back using between 32 and 64 kbps.

[0212] 37. Transcoding OPUS - with probability 0.046, encode to OPUS and back using libopus and between 2 and 64 kbps.

[0213] Appendix D: Considered Approaches This disclosure is compared with nine existing approaches: 1.ITU-P563 (L. Malfait, J. Berger, and M. Kastner, “P.563 - The ITU-T standard for single-ended speech quality assessment,” IEEE Trans. On Audio, Speech and Language Processing, vol. 14, no. 6, pp. 1924-1934, 2010) - This is a reference-free standard designed for narrowband telephony. It was chosen because it was the closest reference-free standard we had access to. The scores generated were used directly.

[0214] 2.FL-JND - Inspired by Manocha et al. (P. Manocha, A. Finkelstein, Z. Jin, NJ Bryan, R. Zhang, and GJ Mysore, “A differentiable perceptual audio metric learned from just noticeable differences,” ArXiv:2001.04460, 2020), the proposed encoder architecture was implemented and trained on the JND task. Then, for each dataset, a small MLP was trained that takes as input the latent features from all encoder layers and predicts the quality score with the sigmoid output.

[0215] 3.FL-PASE - A PASE encoder (S. Pascual, M. Ravanelli, J. Serra A. Bonafonte, and Y. Bengio, “Learning problem-agnostic speech representations from multiple self-supervised tasks,” in Proc. of the Int. Speech Comm. Assoc. Conf. (INTERSPEECH), 2019, pp. 161-165) was trained on the tasks of JND, DT, and speaker identification. Then, for each dataset, a small MLP was trained that takes the latent features from the final layer as input and predicts the quality score with the sigmoid output.

[0216] 4.SRMR (T.H. Falk, C. Zheng, and W.-Y. Chan, “A non-intrusive quality and intelligibility measure of reverberant and dereverberated speech,” IEEE Trans. on Audio, Speech, and Language Processing, vol. 18, no. 7, pp. 1766-1774, 2010) - https: / / github:com / jfsantos / SRMRpy is used and a small MLP with sigmoid output is used to fit it to the corresponding data set.

[0217] 5. AutoMOS (B. Patton, Y. Agiomyrgiannakis, M. Terry, K. Wilson, RA Saurous, and D. Sculley, “AutoMOS: learning a non-intrusive assessor of naturalness-of-speech,” in NIPS16 End-to-end Learning for Speech and Audio Processing Workshop, 2016) - This approach was reimplemented, but the synthesized speech embeddings and their auxiliary losses are L. MR was replaced by.

[0218] 6.Quality-Net (S.-W. Fu, Y. Tsao, H.-T. Hwang, and H.-M. Wang, “Quality-Net: an end-to-end non-intrusive speech quality assessment model based on BLSTM,” in Proc. of the Int. Speech Comm. Assoc. Conf. (INTERSPEECH), 2018, pp. 1873-1877) - The proposed approach is reimplemented.

[0219] 7.WEnets (AA Catellier and SD Voran, “WEnets: a convolutional framework for evaluating audio waveforms,” ArXiv:1909.09024, 2019) - The proposed approach was applied to regression MOS.

[0220] 8.CNN-ELM (H. Gamper, CKA Reddy, R. Cutler, IJ Tashev, and J. Gehrke, “Intrusive and non-intrusive perceptual speech quality assessment using a convolutional neural network,” in IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), 2019, pp. 85-89) - The proposed approach has been reimplemented.

[0221] 9. NISQA (G. Mittag and S. Moeller, “Non-intrusive speech quality assessment for super-wideband speech communication networks,” in Proc. of the IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 7125-7129) - The proposed approach is adapted to work with MOS, and the auxiliary POLQA loss is L MR was replaced by.

Claims

1. 1. A method of training a neural network based system for determining a measure of audio quality of an audio input, the method comprising: obtaining as input at least one training set comprising audio samples, the audio samples comprising a first type of audio samples and a second type of audio samples, each of the first type of audio samples being labelled with information representative of a respective predefined audio quality metric and each of the second type of audio samples being labelled with information representative of a respective audio quality metric relative to that of a reference audio sample; inputting the training set into the neural network-based system; and iteratively training the system to predict label information for each of the audio samples in the training set; wherein the training is based on a plurality of loss functions; and the plurality of loss functions are generated to reflect differences between the label information of the audio samples in the training set and their respective predictions; the first type of audio samples comprises human-annotated audio samples, each of which is labeled with information representative of the respective predetermined audio quality metric, and the second type of audio samples comprises algorithmically generated audio samples, each of which is labeled with information representative of a relative audio quality metric; wherein each of the algorithmically generated audio samples is generated by selectively applying at least one degradation function, each with a respective degradation strength, to a reference audio sample or to an audio sample generated by another algorithm, and wherein the label information includes information indicative of the respective degradation function that has been applied and / or the respective degradation strength.

2. 2. The method of claim 1, wherein the human-annotated audio samples include Mean Opinion Score (MOS) audio samples and / or Just Noticeable Difference (JND) audio samples.

3. The method of claim 1 , wherein the label information further comprises information representative of a degradation relative to other audio samples in the training set or relative to the reference audio sample.

4. 4. The method of claim 1 or 3, wherein the point loss function is selected from a plurality of available point loss functions.

5. 5. The method of claim 4, wherein the plurality of available degradation functions includes functions related to one or more of the following: reverberation, clipping, encoding with a different codec, phase distortion, audio reversal, and background noise.

6. 6. The method according to claim 1, wherein the audio samples generated by the algorithm are arranged in pairs of audio frames {x i ,x j } and / or quadruple audio frames {x ik ,x il ,x jk ,x jl }, and the audio frame x i is generated by selectively applying at least one degradation function, each with a respective degradation strength, to a reference audio frame, said audio frame x j applies at least one degradation function, each with a respective degradation strength, to the audio frame x i , wherein the audio frame x ik ,x il is the time delay for each audio frame x i By selectively applying i The audio frame x is extracted from jk ,x jl is the time delay for each audio frame x j By selectively applying j The method is extracted from

7. 7. The method of claim 1, wherein the loss functions include a first loss function representing a MOS error metric, the first loss function being calculated based on a difference between a MOS ground truth for audio samples in the training set and a prediction for the audio samples.

8. 8. The method according to claim 1, wherein the label information of the second type of audio samples comprises relative information representing whether one audio sample is more degraded than another audio sample, and the loss function comprises a second loss function representing a pairwise ranking metric, the second loss function being calculated based on a ranking set by label information comprising the relative degradation information and a prediction thereof.

9. 9. The method of claim 8, wherein the system is trained in such a way that one audio sample that is less degraded obtains an audio quality metric that represents better audio quality than another audio sample that is more degraded.

10. 8. The method according to claim 1, wherein the label information of the second type of audio samples comprises relative information representing perceptual relevance between audio samples, and the loss function comprises a third loss function representing a consistency metric, the third loss function being calculated based on a difference between the label information comprising the information of the perceptual relevance and its prediction.

11. The method of claim 10, wherein the consistency metric indicates whether two or more audio samples have the same degradation function and degradation strength and correspond to the same time frame.

12. 8. The method according to claim 1, wherein the label information of the second type of audio samples comprises relative information representing whether an audio sample is applied with the same degradation function and the same degradation strength as another audio sample, and the loss function comprises a fourth loss function representing a degradation state metric, the fourth loss function being calculated based on a difference between the label information comprising the relative degradation information and the prediction thereof.

13. 13. The method of claim 1, wherein the label information of the second type of audio samples comprises relative information representing a perceptual difference with respect to one another, and the loss function comprises a fifth loss function representing a JND metric, the fifth loss function being calculated based on a difference between the label information comprising the relative perceptual difference and its prediction.

14. 14. The method according to claim 1, wherein the label information of the second type of audio samples comprises information representative of a degradation function applied to the audio samples, and the loss functions comprise a sixth loss function representative of a degradation type metric, the sixth loss function being calculated based on a difference between the label information comprising the information of the respective degradation function and its prediction.

15. 15. The method according to claim 1, wherein the label information of the second type of audio samples comprises information representative of a degradation strength applied to the audio samples, and the loss function comprises a seventh loss function representative of a degradation strength metric, the seventh loss function being calculated based on a difference between the label information comprising the respective degradation strength information and its prediction.

16. 16. The method of claim 1, wherein the loss function comprises an eighth loss function representing a regression metric, the regression metric being calculated according to at least one of a reference-based quality measure and a non-reference based quality measure.

17. 17. The method of claim 16, wherein the reference-based quality measures include at least one of PESQ, CSIG, CBAK, COVL, SSNR, LLR, WSSD, STOI, SISDR, Mel-Cepstral Distortion, and Log-Mel-Band Distortion.

18. 18. The method of claim 1, wherein each of the audio samples in the training set is used in at least one of the plurality of loss functions, and a final loss function for the training is generated based on an averaging process for one or more of the plurality of loss functions.

19. 19. The method of any one of claims 1 to 18, wherein the system includes an encoding stage for mapping the audio input into a feature space representation, and an evaluation stage for generating a prediction of label information based on the feature space representation.

20. 20. The method of claim 19, wherein the encoding stage for generating the feature space representation includes a neural network encoder.

21. 21. The method of claim 1, wherein each of the plurality of loss functions is determined based on a neural network including a linear layer or a multi-layer perceptron (MLP).

22. 1. A neural network based system for determining an audio quality measure of input audio samples, comprising: An encoding stage; and Evaluation stage: the encoding stage configured to map the input audio samples to a feature space representation; and The evaluation stage is adapted to predict information representative of a predetermined audio quality metric based on the feature space representation, and further to predict information representative of a relative audio quality metric to a reference audio sample; the system further comprising: obtaining as input at least one training set, said training set including audio samples of a first type and audio samples of a second type, each of said first type being labeled with information representative of a respective predefined audio quality metric, and each of said second type being labeled with information representative of a respective audio quality metric relative to that of a reference audio sample; inputting the training set into the system; and iteratively training the system based on the training set to predict label information for each of the audio samples in the training set based on a plurality of loss functions, the plurality of loss functions being generated to reflect differences between the label information of the audio samples in the training set and their respective predictions; configured to execute; the first type of audio samples comprises human-annotated audio samples, each of which is labeled with information representative of the respective predetermined audio quality metric, and the second type of audio samples comprises algorithmically generated audio samples, each of which is labeled with information representative of a relative audio quality metric; wherein each of the algorithmically generated audio samples is generated by selectively applying at least one degradation function, each with a respective degradation strength, to a reference audio sample or to an audio sample generated by another algorithm, and the label information includes information indicative of the respective degradation function and / or the respective degradation strength that has been applied.

23. A method of operating a neural network based system for determining an audio quality measure of input audio samples after the method of claim 1 has been performed, said system including an encoding stage and an evaluation stage, the method comprising: mapping, by the encoding stage, the input audio samples into a feature space representation; and predicting, by said evaluation stage, information representative of a predetermined audio quality metric and information representative of a relative audio quality metric with respect to a reference audio sample based on said feature space representation; The method includes:

24. 24. A program comprising instructions which, when executed by a processor, cause the processor to perform the steps of the method according to any one of claims 1 to 21 and 23.

25. A computer-readable storage medium storing the program according to claim 24.

26. An apparatus comprising a processor and a memory coupled to the processor, the processor configured to cause the apparatus to perform the steps of the method according to any one of claims 1 to 21 and 23.

Images