Method for learning audio quality indicators in combination with labeled data and unlabeled data

Abstract

A method of training a neural network-based system for determining an indication of audio quality of an audio input is described. The method comprises obtaining, as input, at least one training set comprising audio samples. The audio samples comprise audio samples of a first type and audio samples of a second type, wherein each of the audio samples of the first type is labeled with information indicative of a respective predetermined audio quality indicator, and wherein each of the audio samples of the second type is labeled with information indicative of a respective audio quality indicator relative to an audio quality indicator of a reference audio sample. The method further comprises inputting the training set into the neural network-based system, and iteratively training the system to predict respective label information of the audio samples in the training set.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to the following prior applications: ES application 202030605 (reference number: D20045ES), filed June 22, 2020; U.S. Provisional Application 63 / 072,787 (reference number: D20045USP1), filed August 31, 2020; U.S. Provisional Application 63 / 090,919 (reference number: D20045USP2), filed October 13, 2020; and EP application 20203277.7 (reference number: D20045EP), filed October 22, 2020, all of which are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to the field of audio processing. Specifically, this disclosure relates to techniques for speech / audio quality assessment using machine learning models or systems, and to frameworks for training machine learning models or systems for speech / audio quality assessment. Background Technology

[0004] Speech or audio quality assessment is crucial for countless research topics and real-world applications. Its requirements range from algorithm evaluation and development to basic analysis or informed decision-making. Broadly speaking, audio quality assessment can be performed through subjective auditory tests or objective quality metrics. Objective metrics, closely related to human judgment, offer the possibility of extending automated quality assessment, yielding consistent results with only a fraction of the effort, time, and cost of subjective metrics. Traditional objective metrics rely on standard signal processing blocks such as the Short-Time Fourier Transform or perceptual stimulus blocks such as gamma-pass filter banks. These, along with further processing blocks, create often intricate rule-based systems. An alternative approach is to learn speech quality directly from raw data by combining machine learning techniques with carefully selected stimuli and their corresponding human ratings. Rule-based systems may have the advantage of perceptual stimulus and are interpretable to some extent, but often focus only on specific types of signals or degradations, such as telephone signals or Voice over IP (VoIP) degradation. On the other hand, learning-based systems are often easily reusable to other tasks and degradations, but require large amounts of manually annotated data. In addition, both rule-based and learning-based systems may lack generalization ability and therefore perform poorly on out-of-sample but still-focused data.

[0005] Therefore, there is a need for methods and systems to perform (automatic) audio quality assessment, and possibly also for methods to train such systems for (automatic) audio quality assessment so that improved performance (e.g., in error rate, consistency, etc.) and / or efficiency can be achieved, while allowing good generalization to new audio (e.g., recordings) and / or listeners. Summary of the Invention

[0006] In view of the foregoing, this disclosure generally provides a method for training a neural network-based system for determining an indication of the audio quality of an audio input, a neural network-based system for determining an indication of the audio quality of an input audio sample, and a method for operating a neural network-based system for determining an indication of the audio quality of an input audio sample, as well as corresponding programs, computer-readable storage media, and apparatus having the features of the respective independent claims. The dependent claims relate to preferred embodiments.

[0007] According to one aspect of this disclosure, a method is provided for training a deep learning-based (e.g., neural network-based) system for determining an indication of the audio quality of an audio input. Training may refer to determining parameters of the deep learning model(s)(s) (e.g., neural networks(s)) used to implement the system. Further, training may refer to iterative training. For example, the indication of the audio quality of the audio input may be a score. If desired, the score may be normalized (restricted) to a predetermined range, such as between 1 and 5. The method may include obtaining at least one training set comprising audio samples as input(s). In particular, the audio samples may include audio samples of a first type and audio samples of a second type. More specifically, each of the audio samples of the first type may be labeled with information indicating a corresponding predetermined audio quality metric (e.g., between 1 and 5), and each of the audio samples of the second type may be labeled with information indicating a corresponding audio quality metric relative to a reference audio sample (e.g., relative to an audio quality metric of another audio sample in the training set). In other words, the audio samples of the first type can be considered to each include label information indicating an absolute audio quality metric (e.g., normalized between 1 and 5, where 5 is the highest audio quality). In contrast, the second type of audio sample can be considered as each including label information indicating a relative audio quality metric. As those skilled in the art will understand and recognize, the reference audio sample used herein can be, but does not necessarily have to be, another audio sample in the training set. In other words, the reference audio sample can be an external reference audio sample (i.e., not in the training set) or an internal reference audio sample (i.e., within the training set). Furthermore, the reference audio sample can be any suitable audio sample, for example, a predefined or predetermined audio sample, which can serve as a (comparative) reference such that, in a general sense, a relative metric can be determined (e.g., calculated) by comparing the audio sample with the reference audio sample. In some examples, the relative label information may include information indicating that the audio sample degrades more (or less) than the (predetermined) reference audio sample (e.g., another audio sample in the training set). In some examples, the relative label information may include information indicating a particular degradation function (and optionally, the corresponding degradation intensity) that has been applied, for example, to the reference audio sample (e.g., another audio sample in the training set) when generating the (degraded) audio sample. Of course, as those skilled in the art will understand and recognize, any other suitable relative label information may be included if desired or appropriate. The method may further include inputting a training set into a deep learning-based system and iteratively training the system to predict the corresponding label information of audio samples in the training set. Training may be based on multiple loss functions. Specifically, the multiple loss functions may be generated to reflect the difference between the label information of the audio samples in the training set and their corresponding predictions.

[0008] As configured as described above, the proposed method, broadly speaking, can train a neural network to generate non-invasive quality ratings. Because the ratings are learned from data, the focus can be reused by changing the type of audio used for training, and degradations of interest can also be selected for learning. Notably, the proposed method is generally semi-supervised, meaning it can utilize both absolute and relative ratings obtained from different data sources. This reduces the need for expensive and time-consuming listener data. In addition to learning from multiple sources, the proposed method also learns from multiple features of those sources by training the network based on multiple loss functions (generated from audio samples in the data sources), thus leading to more general automated measurement.

[0009] In some examples, the first type of audio samples may include manually annotated audio samples. Each manually annotated audio sample may be labeled with information indicating a corresponding predetermined audio quality metric. As can be understood and recognized by a technician, audio samples may be annotated in any suitable manner, such as by audio experts, general listeners, mechanical turkers (e.g., crowdsourcing), etc.

[0010] In some examples, the manually annotated audio samples may include Mean Opinion Score (MOS) audio samples and / or Minimum Perceptible Difference (JND) audio samples. Sections B.1 and B.2 of the appendix provide some possible examples of the MOS and JND datasets, respectively.

[0011] In some examples, the second type of audio samples may include audio samples generated by an algorithm (or programmed, human), each audio sample being labeled with information indicating a relative audio quality metric.

[0012] In some examples, each algorithm-generated sample can be generated by selectively applying at least one degradation function, each with a corresponding degradation intensity, to a reference audio sample or an audio sample generated by another algorithm. In such examples, labeling information may include information indicating the corresponding degradation function and / or corresponding degradation intensity that has been applied to the reference audio sample or the audio sample generated by the other algorithm. Of course, as those skilled in the art will recognize, any other suitable algorithm and / or program can be used to generate the second type of audio samples.

[0013] In some examples, the label information may further include information indicating degradation relative to each other. That is, in some examples, the label information may further include information indicating degradation relative to a reference audio sample or other audio samples in the training set. For example, the label information may include relative information indicating that one audio sample degrades relatively more or less than another audio sample (e.g., an external reference audio sample or another audio sample in the training set).

[0014] In some examples, the degradation function can be selected from multiple available degradation functions. For example, multiple available degradation functions can be implemented as a degradation function pool. Alternatively or alternatively, the corresponding degradation strength can be set such that degradation at its minimum may still be perceptible (e.g., perceptible to an expert, audience, or author).

[0015] In some examples, the multiple available degradation functions may include functions associated with one or more of the following: reverberation, clipping, encoding with different codecs, phase distortion, audio inversion, and background noise. Further, the (background) noise may include real (e.g., recorded) background noise or artificially generated background noise. Note that in some cases, the selected degradation intensity may only be one aspect of the overall degradation, and for other relevant aspects, values ​​may be randomly sampled between empirically chosen values. For example, in the case of reverberation effects, the signal-to-noise ratio (SNR) may be chosen as the primary intensity, but the reverberation type, width, delay, etc., may also be randomly selected. Part C of the appendix provides some possible examples of degradation and / or intensity.

[0016] In some examples, the audio samples generated by the algorithm can be generated as audio frame pairs {x} i x j} and / or audio frame quadruple Specifically, audio frame x i Audio frames can be generated by selectively applying (e.g., externally) at least one degradation function, each with a corresponding degradation intensity, to a reference audio frame (or audio frames in the training set). Then, the audio frame x... j At least one degradation function, each with a corresponding degradation intensity, can be selectively applied to audio frame x. i To generate. Further, audio frames. and This can be achieved by selectively applying a corresponding time delay to audio frame x. i From audio frame x i Extract from, and audio frames and This can be achieved by selectively applying a corresponding time delay to audio frame x. j From audio frame xj Extracted from. Audio frame x is shown as an example, not a limitation. i The length can be 1.1 seconds, and it can be calculated from 1.1-second audio frames x i Audio frames extracted and The length can be 1 second. As can be understood and recognized by those skilled in the art, audio samples can be generated in any suitable manner, depending on various implementation methods and / or requirements.

[0017] In some examples, the loss function may include a first loss function that indicates a MOS error metric. The first loss function can be calculated based on the difference between the true MOS data for audio samples in the training set and the predictions for the audio samples. In this sense, the first loss function can also be considered as an indicator of the MOS opinion score metric in some cases. Of course, as those skilled in the art will understand and recognize, in addition to the difference, any other suitable means (such as appropriate mathematical concepts like divergence or cross-entropy) can be used to determine (calculate) the first loss function (or any other suitable loss function that will be discussed in detail below).

[0018] In some examples, the label information for the second type of audio samples may include relative (label) information indicating whether one audio sample degrades more (or, in some cases, less) than another audio sample. In addition to or instead of the first loss function as shown above, an additional loss function may include a second loss function indicating a pairwise ranking metric. Specifically, the second loss function may be calculated based on a ranking established by including relative degradation information and its predicted label information.

[0019] In some examples, the system can be trained in such a way that an audio sample with less degradation obtains an audio quality metric indicating better audio quality than another audio sample with more degradation.

[0020] In some examples, the label information for the second type of audio samples may include relative information indicating the perceptual relevance between audio samples. For example, perceptual relevance can indicate the perceptual difference or perceptual similarity between two audio samples or between two pairs of audio samples. That is, broadly speaking, if two audio signals are extracted from the same (audio) source and differ by only a few audio samples, or if the difference between the two signals is perceptually uncorrelated, then their corresponding quality metrics (or quality scores) should be substantially the same. Additionally, if two signals are perceptually distinguishable, then their metric / score difference should be above a certain bound. It is worth noting that, for example, these two concepts can also be extended to pairs by considering the consistency between pairs of score differences. Therefore, the loss function may additionally or alternatively include a third loss function indicating a consistency metric, and in particular, the third loss function may be calculated based on the difference between the label information, which includes perceptual relevance information, and its prediction. In this sense, the third loss function can also be considered as an indicator of score consistency in some cases.

[0021] In some examples, consistency metrics can indicate whether two or more audio samples have the same degradation function and / or degradation intensity and whether they correspond to the same time frame.

[0022] In some examples, the label information for the second type of audio samples may include relative information indicating whether an audio sample has been applied the same degradation function and the same degradation intensity as another audio sample. Therefore, the loss function may additionally or alternatively include a fourth loss function indicating (the same or different) degradation condition metrics. Specifically, the fourth loss function may be calculated based on the difference between the label information, which includes relative degradation information / conditions, and its prediction.

[0023] In some examples, the label information for the second type of audio samples may include relative information indicating perceptual differences relative to each other. Therefore, the loss function may additionally or alternatively include a fifth loss function indicating the JND metric, and this fifth loss function may be calculated based on the difference between the label information, which includes relative perceptual differences, and its prediction.

[0024] In some examples, the label information for the second type of audio samples may include information indicating the degradation function that has been applied to the audio samples. Therefore, the loss function may additionally or alternatively include a sixth loss function indicating the degradation type. Specifically, the sixth loss function may be calculated based on the difference between the label information, which includes the corresponding degradation function type information, and its prediction.

[0025] In some examples, the label information for the second type of audio samples may include information indicating the degradation intensity that has been applied to the audio samples. Therefore, the loss function may additionally or alternatively include a seventh loss function indicating the degradation intensity metric. And the seventh loss function may be calculated based on the difference between the label information, including the corresponding degradation intensity information, and its prediction.

[0026] In some examples, the loss function may additionally or alternatively include an eighth loss function indicating the regression metric. In particular, the regression metric may be calculated based on at least one of a reference-based quality metric and / or a referenceless quality metric.

[0027] In some examples, reference-based quality metrics may include, but are not limited to, at least one of the following: perceptual evaluation of speech quality (PESQ), composite signal measurement (CSIG), composite noise measurement (CBAK), composite overall quality measurement (COVL), segmented signal-to-noise ratio (SSNR), log-likelihood ratio (LLR), weighted slope spectral distance (WSSD), short-term objective intelligibility (STOI), scale-invariant signal-to-distortion ratio (SISDR), Mel-Cepstral distortion, and log-Mel band distortion. Of course, as those skilled in the art will recognize, any other suitable reference-based quality metric and / or reference-free quality metric may be used.

[0028] In some examples, each audio sample in the training set can be used in at least one of multiple loss functions. That is, some audio samples in the training set can be reused or shared by one or more loss functions. For example, an (algorithm-generated) audio sample used to compute the third loss function (i.e., the score consistency metric) can be reused when computing the fourth loss function (i.e., the same / different degradation condition metric), and vice versa. This can significantly improve the efficiency of the training system. In particular, the final loss function used for training can be generated based on an averaging process of one or more of the multiple loss functions. As those skilled in the art will recognize, any other suitable means or process can be used to generate a final loss function based on any number of suitable loss functions, depending on various implementation methods and / or requirements.

[0029] In some examples, the system may include an encoding level (or simply an encoder) for mapping (e.g., transforming) the audio input to a feature space representation. For example, the feature space representation may be a latent space (features). The system may then further include an evaluation level for generating predictions of label information based on the feature space representation.

[0030] In some examples, the encoding level used to generate intermediate representations may include a neural network encoder.

[0031] In some examples, each of the multiple loss functions can be determined based on a neural network that includes linear layers or multilayer perceptrons (MLPs).

[0032] According to another aspect of this disclosure, a deep learning-based (e.g., neural network-based) system is provided for determining an indication of the audio quality of an input audio sample. The system can be trained according to any of the examples shown above. In particular, the system may include an encoding level and an evaluation level. More specifically, the encoding level may be configured to map the input audio sample to a feature space representation. Further, the evaluation level may be configured to predict information indicating a predetermined audio quality metric based on the feature space representation, and further predict information indicating a relative audio quality metric relative to a reference audio sample. As those skilled in the art will understand and recognize, the reference audio sample used herein may be, but is not necessarily, another audio sample in the training set used to train the system. In other words, the reference audio sample may be an external reference audio sample (i.e., not in the training set) or an internal reference audio sample (i.e., within the training set). Furthermore, the reference audio sample may be any suitable audio sample, e.g., a predefined or predetermined audio sample, which can serve as a (comparative) reference such that, in a general sense, a relative metric can be determined (e.g., computed) by comparing the audio sample with the reference audio sample. Furthermore, the predictive information (e.g., information indicating a relative audio quality metric relative to a reference audio sample) can be used to further train (adjust) the system.

[0033] In some examples, the system can be configured to take at least one training set as input. Specifically, the training set may include audio samples of a first type and audio samples of a second type, wherein each of the audio samples of the first type is labeled with information indicating a corresponding predetermined audio quality metric, and wherein each of the audio samples of the second type is labeled with information indicating a corresponding audio quality metric relative to a reference audio sample or relative to an audio quality metric of another audio sample in the training set. Further, the system can be configured to input the training set into the system; and to iteratively train the system based on the training set to predict corresponding label information for audio samples in the training set based on a plurality of loss functions, the plurality of loss functions being generated to reflect the difference between the label information of the audio samples in the training set and their corresponding predictions.

[0034] According to another aspect of this disclosure, a method is provided for operating a deep learning-based (e.g., neural network-based) system for determining an indication of the audio quality of an input audio sample. The system may correspond to any of the example systems shown above; and the system may be trained according to any of the example methods shown above. For example, the system may include an encoding level and an evaluation level. In particular, the method may include mapping the input audio sample to a feature space representation by the encoding level. The method may further include the evaluation level predicting information indicating a predetermined audio quality metric and information indicating a relative audio quality metric relative to a reference audio sample based on the feature space representation. As those skilled in the art will understand and recognize, the reference audio sample used herein may be, but does not necessarily have to be, another audio sample in the training set. In other words, the reference audio sample may be an external reference audio sample (i.e., not in the training set) or an internal reference audio sample (i.e., within the training set). Furthermore, the reference audio sample may be any suitable audio sample, for example, a predefined or predetermined audio sample, which can serve as a (comparative) reference such that, in a general sense, the relative metric can be determined (e.g., computed) by comparing the audio sample with the reference audio sample. Furthermore, the predictive information (e.g., information indicating a relative audio quality metric relative to a reference audio sample) can be used to further train (adjust) the system.

[0035] According to a further aspect of this disclosure, a computer program is provided. The computer program may include instructions that, when executed by a processor, cause the processor to perform all the steps of the example methods described throughout the disclosure.

[0036] In a further aspect, a computer-readable storage medium is provided. The computer-readable storage medium can store the aforementioned computer program.

[0037] According to a further aspect, an apparatus is provided comprising a processor and a memory coupled to the processor. The processor may be adapted to cause the apparatus to perform all steps of the example methods described throughout the disclosure.

[0038] It should be recognized that system features and methodological steps can be interchanged in many ways. In particular, as those skilled in the art will recognize, the details of the disclosed (multiple) methods can be implemented by the corresponding system, and vice versa. Furthermore, any of the above statements regarding the (multiple) methods are to be understood as equally applicable to the corresponding system, and vice versa. Attached Figure Description

[0039] The following explanation of exemplary embodiments of this disclosure is based on the accompanying drawings, in which:

[0040] Figure 1AThis is a schematic block diagram of a system for audio quality assessment according to embodiments of the present disclosure.

[0041] Figure 1B This is a schematic diagram of another block diagram of a system for audio quality assessment according to embodiments of the present disclosure.

[0042] Figure 2 This is a flowchart illustrating an example of a method for training a deep learning-based system to determine an indication of the audio quality of an audio input, according to an embodiment of the present disclosure.

[0043] Figure 3 This is a flowchart illustrating an example of a method of a deep learning-based system for determining an indication of the audio quality of an input audio sample, according to embodiments of the present disclosure.

[0044] Figure 4 Figures 8 to 8 are example illustrations showing various results and comparisons based on embodiments of the present disclosure. Detailed Implementation

[0045] The accompanying drawings (Figures) and the following description relate to preferred embodiments only by way of illustration. It should be noted that, based on the following discussion, alternative embodiments of the structures and methods disclosed herein will readily be considered as feasible alternatives without departing from the claimed principles.

[0046] Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying drawings. Note that similar or identical reference numerals may be used in the drawings, and they may indicate similar or identical functions, whenever feasible. The drawings depict embodiments of the disclosed systems (or methods) for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.

[0047] Generally, quality ratings are crucial in the audio industry, used for everything from monitoring channel distortion to developing new processing algorithms. Traditionally, quality ratings are obtained from either general or expert listeners, requiring significant investment in terms of money, time, and infrastructure. This disclosure proposes an automated tool for providing such quality ratings.

[0048] The purpose of automated tools (or algorithms) for measuring audio quality is to obtain a reliable proxy for human ratings, thus addressing the input problem mentioned above. Several automated tools exist for measuring the speech quality of audio files. Given some input audio, such tools produce scores associated with some subjective audio quality rating, typically between 1 and 5.

[0049] One difference between these tools is whether they use reference (clean) audio for comparison (invasive vs. non-invasive). Another difference is whether they are handcrafted / predefined or learned from data. Further considerations include the range of audio to be analyzed and the specific degradation or distortion that will be detected in the metric.

[0050] Therefore, a key driving factor for this disclosure is the recognition that additional evaluation criteria / tasks should be considered beyond conventional or reasonable measures of speech quality, such as Mean Opinion Score (MOS). Specifically, it was decided to draw upon such additional evaluation criteria. Another fundamental aspect of this disclosure is the recognition that further objectives, datasets, and tasks exist that can complement those criteria and aid in learning more robust representations of speech quality and scores.

[0051] In light of this, this disclosure proposes, broadly speaking, a method for training a neural network to generate non-invasive quality ratings. Because the ratings are learned from data, the focus can be reused by changing the type of audio used to train the neural network, and the degradation of interest can also be selected for learning. Notably, the proposed method is generally semi-supervised, meaning that it can utilize ratings obtained from human listeners (e.g., embedded in manually annotated data, sometimes also called labeled data) and raw (unrated) audio (sometimes also called unlabeled data) as input data. This reduces the need for expensive and time-consuming listener data. In addition to learning from multiple sources, the proposed method also learns from multiple representations of those sources, thus leading to more general automated measurements. Additional design principles of the proposed method (and system) may include, but are not limited to, lightweight and fast operation that is inherently fully differentiable, and the ability to process short raw audio frames, such as 48 kHz (thus generating time-varying dynamic estimates).

[0052] refer to Figure 1A This diagram illustrates a simplified block diagram of a system 100 for audio quality evaluation according to an embodiment of the present disclosure. System 100 may consist of an encoding level (or simply encoder) 1010 and an evaluation level 1020. Figure 1A As shown in the example, evaluation level 1020 may include a series of “headers” 1021, 1022, and 1023, sometimes (uniformly) denoted as H. See below for reference. Figure 1BThe different heads are described in detail. Broadly speaking, each head can be considered a separate computational unit suitable for determining the corresponding label information (e.g., absolute or relative quality metrics) associated with a given audio sample (frame). Typically, encoder 1010 can take the raw input audio signal (e.g., audio frame) x 1000 and map (or transform) it to, for example, a latent space representation (vector) z 1005. Different heads can then take these latent vectors z 1005 and compute the output against one or more considered criteria (exemplarily shown as 1025). It is worth noting that in some cases, when processing {z... i , z j When}, the head can concatenate (or any other suitable form) it as input.

[0053] In some examples, encoder 1010 may include four main levels, such as Figure 1A As shown. First, encoder 1010 can transform the distribution of x 1000 by applying a μ-law formula with learnable μ (e.g., without quantization). Generally, the μ-law algorithm (sometimes written as "μ-law") is a companding algorithm, primarily used in, for example, 8-bit PCM digital telecommunications systems. Notably, companding algorithms can be used to reduce the dynamic range of audio signals. In analog systems, this can improve the SNR achieved during transmission; in the digital domain, it can reduce quantization errors (thus increasing the signal-to-quantization-noise ratio). For example, the value of μ might be initialized to 8 at the beginning. Next, block 1001 can be employed, which in some examples may include a series (e.g., 4) pooling sub-blocks composed of convolution, batch normalization (BN), rectified linear unit (ReLU) activation, BlurPool, or any other suitable block / module. As an example and not a limitation, 32, 64, 128, and 256 filters with a kernel width of 4 and a downsampling factor of 4 can be used. Of course, any other suitable implementation can be employed, as those skilled in the art will recognize. For example, possible alternatives to convolution include, but are not limited to, linear layers, recurrent neural networks, attention modules, or transformers. Possible alternatives to batch normalization include, but are not limited to, layer normalization, instance normalization, or group normalization. In some other implementations, batch normalization may be omitted entirely. Possible alternatives to ReLU include, but are not limited to, sigmoid gates, tanh gates, gated linear units, parameterized ReLU, or leaky ReLU. Possible alternatives to BlurPool include, but are not limited to, strided convolution, max pooling, or average pooling. It should be further understood that, as those skilled in the art will recognize, the foregoing alternative implementations can be combined with each other as needed or where feasible.

[0054] Next, block 1002 can be used. In some examples, this block may include multiple (e.g., 6) residual blocks formed by BN pre-activation, followed by 3 blocks of ReLU, convolution, and BN. As an example, and not a limitation, 512, 512, and 256 filters with kernel widths of 1, 3, and 1 can be used, connected via parametrically linearly averaged residual connections: h′=a′h+(1-a′)F(h), where a′=σ(a) is a vector of learnable parameters between 0 and 1, and F is the residual network (e.g., all components of a can be initialized to 3, such that training primarily begins with a bypass from h to h'). After residual block 1002, temporal statistics can be computed in block 1003, such as taking the mean and standard deviation for each channel. This step aggregates all temporal information into a single vector (e.g., 2×256 dimensions). Subsequently, in block 1004, a batch normalization (BN) operation can be performed on such a vector, and then it can be fed into a multilayer perceptron (MLP), which is formed, for example, by two linear layers with BN and ReLU activation in the middle. As an example and not a limitation, 1024 and 200 units can be used.

[0055] Now for reference Figure 1B The diagram shows a more detailed block diagram of a system 110 for audio quality assessment according to an embodiment of the present disclosure. It is worth noting that... Figure 1B The same or similar reference numerals in system 110 indicate, for example Figure 1A The same or similar elements in the system 100 shown are omitted for the sake of brevity; therefore, repeated descriptions of them can be omitted. Specifically, in... Figure 1B In the example system 110, the focus will be on the evaluation level 1120, where the different learning / training criteria of the head will be discussed in detail below.

[0056] refer to Figure 1B System 110, broadly speaking, can train a (convolutional) neural network that can transform an audio input x 1100 into a (low-dimensional) latent space representation z 1105 and later output a single-value score s1140. Similar to... Figure 1A As shown, the network / system can be formed by two main blocks (levels): an encoding level (or sometimes called an encoder network) 1110 that outputs the latent vector z 1105, and an evaluation level 1120 that includes multiple different "heads" that further process the latent vector z 1105. Notably, one of the heads is responsible for generating the final score s 1140, while the remaining heads are typically used to adjust the latent space (they can also be used as predictors for the quantities used to train it).

[0057] Similar to Figure 1AThe coding level 1110 can employ a μ-law logarithmic representation of the audio and pass it through a series of convolutional blocks. For example, firstly, multiple BlurPool blocks (e.g., 1101) can decimate the signal to a lower time span. Next, multiple ResNet blocks (e.g., 1102) can further process the obtained representation. Then, temporal statistics such as mean, standard deviation, minimum, and maximum values ​​(e.g., 1103) can be used to summarize the audio frames. Finally, an MLP (e.g., 1104) can be used to perform the mapping between those statistics and z-values ​​1105.

[0058] Different heads can take a z vector of 1105 and predict different quantities 1121-1128. Generally, during training, each head can have a loss function that imprints the desired features onto the score s of 1140 or the latent space z of 1105.

[0059] It is worth noting that, as a technician will recognize, the score s can be calculated in any suitable manner. Some possible examples of how the score s can be calculated are provided, for example, in Part A of the Appendix.

[0060] Now for reference Figure 1B For System 110, examples of various possible learning or evaluation criteria and their corresponding loss functions corresponding to possible heads will be discussed in detail below. In some cases, some of these criteria can be considered auxiliary tasks. In other words, not all criteria must be used when training the system, and some criteria can be omitted or ignored depending on various implementation methods and / or requirements. Of course, as those skilled in the art will understand and recognize, the criteria (or heads) are not limited to those discussed herein, but can be extended or adapted to any particular situation.

[0061] Average opinion score

[0062] The primary and almost sole criterion considered by traditional methods is likely MOS error. In some cases, this can also be simply referred to as a score head 1121. Generally, this score head takes z 1105 as input and passes it through, for example, a linear layer (or an MLP or any other suitable neural network) 1131 to produce a single quality score value s. As an example, this score can be bounded by a sigmoid function and rescaled, for example, between 1 and 5 (e.g., 5 is the highest quality). And to compute the loss of this head, for example, ratings provided by human listeners (if available) can be used. An alternative could be to use ratings provided by other existing quality metrics (reference-based or no-reference). In other words, broadly speaking, the loss function can be considered to include a first loss function indicating the MOS error metric, and this first loss function can be computed based on the difference between the true MOS data of audio samples in the training set and the predictions for the audio samples.

[0063] More specifically (but not as a limitation), learning-based methods typically include a supervised regression problem, such that...

[0064]

[0065] in, 1141 is the actual MOS data, s i It is the score predicted by the model, and |||| corresponds to a certain norm. For example, the L1 norm (mean absolute error) or any other suitable norm can be used.

[0066] In one example, system 110 can predict the latent representation z by using, for example, linear units and sigmoid activation σ. i Score s i :s i =1+4σ(w T z i +b), where example coefficients 1 and 4 make the sample suitable for MOS values ​​between 1 and 5. Latent representation Z i The original audio frame x can be processed using a neural network encoder 1110. i It is obtained through encoding.

[0067] Pair sorting

[0068] Besides MOS, another intuitive but often overlooked concept in quality assessment might be pairwise ranking. In some cases, this can also be simply referred to as the ranking head 1122. Generally, this pairwise ranking head 1122 takes score pairs (e.g., s1 and S2) as input, which can be obtained from a previous scoring head after processing audios x1 and x2. It can then use flags (such as tag information) to calculate a ranking-based loss, which signals which audio degrades more (or less) (if available). For example, if x1 degrades / distorts more than x2 (or vice versa), the loss might cause S1 to be lower than S2. In other words, broadly speaking, the loss function can be considered to include a second loss function indicating the pairwise ranking metric, and this second loss function can be calculated based on the difference between the tag information (e.g., the ranking established by the tag information) including relative degradation information and its prediction.

[0069] More specifically (but not as a limitation), under the concept of pairwise ordering, if the speech signal x j It is the same (originally "pure" or "purer") word x i If the programming (algorithm) is a degenerate version, then its score should reflect this relationship, i.e., s i ≥s j This concept can then be introduced into the training scheme by considering a learning ranking strategy. In one example, it could follow the bounded loss formula.

[0070] L RANK =max(0, s) j -s i +α) (2)

[0071] Here, α = 0.3 (or any other suitable value) can be used as a boundary constant.

[0072] In one example, for {x i x j}1142 can be generated programmatically by considering multiple datasets with “clean” speech (or reference speech) and a pool of several degradation functions.

[0073] For {x i x j}1142 can be generated in any suitable manner. As an example, and not a limitation, to form each pair, the following can be done:

[0074] • Perform uniform sampling on the dataset, and then perform uniform sampling on the files within it.

[0075] • Sample frames uniformly for 1.1 seconds (or any other suitable length) to avoid silent frames or predominantly silent frames. Normalize them so that the maximum absolute amplitude is 1.

[0076] • With probabilities of 0.84, 0.12, and 0.04, sample zero, one, or two degradations from the available degradation pool (discussed in detail later). If it is a zero degradation, the signal directly becomes x. i Alternatively, the intensity of each degradation can be uniformly selected and applied sequentially to generate x. i .

[0077] • Sample one, two, three, or four degradations from the available degradation pool with probabilities of 0.75, 0.2, 0.04, and 0.01. Uniformly select the intensity and apply it sequentially to x. i To generate x j .

[0078] It should be understood that the above embodiments (including the probabilities mentioned) are for illustrative purposes only and are not intended to be limiting. As those skilled in the art will recognize, any other suitable probabilities or embodiments may be applied herein.

[0079] Then, the generated pair {x i x j It can be stored together with degradation type and / or intensity information (e.g., stored as tag information).

[0080] Further information about possible methods for generating pairs can be found, for example, in section B.3 of the appendix.

[0081] Alternatively or alternatively, random pairs can be collected from, for example, (artificially) annotated data, thereby based on, for example, the corresponding s * Assign indices i and j such that s has a larger value. * The element of a pair can be obtained at index i, and vice versa. For pairs from annotation data, the bounding constant can be set, for example, to... Or any other suitable value.

[0082] Scoring consistency

[0083] Consistency is another often overlooked concept in audio quality assessment. Generally, a consistency head 1123 can take score pairs s1 and s2 corresponding to audios x1 and x2, respectively, as input. It can then use flags (e.g., label information) to calculate a distance-based loss, which signals whether the audios are likely to have the same type and / or level of degradation (if available). For example, if x1 and x2 have the same distortion / degradation and are at the same level, the loss might encourage s1 to be closer to s2 (in some cases, it can be assumed, if needed, that similar original content exists in x1 and x2). It might also encourage similar implementations of x1 and x2 with different degradations x′1 and x′2 to be closely linked together (e.g., x1 with x′1 and x2 with x′2). In other words, broadly speaking, the loss function can be considered to include a third loss function indicating a consistency metric, and this third loss function can be calculated based on the difference between the label information, including perceived relevance information, and its prediction.

[0084] More specifically (but not as a limitation), under the concept of score consistency, if two signals x k and x l Extracted from (essentially) the same source and differing by only a few audio samples, or if two signals x k and x l If the differences between them are perceptually unrelated, then their scores should be essentially the same, i.e., s k =s l Additionally, if two signals x i and x j If the differences are perceptually distinguishable, then the score difference should be above a certain (e.g., a predetermined) threshold, i.e., |s i -s j |≥β. Note that these two concepts can be further extended to, for example, paired pairs by considering the consistency between pairs of score differences. In one possible implementation, the first concept can be extended to: if there are perceptually related pairs of x... il and x jl Two identical signals x ik and x jk (where x) j Compared to x i With more degradation (from which signals k and l are extracted), the score differences should tend to be equal, i.e., s ik -s jk =s il -s jl .

[0085] In one example, if all three concepts mentioned above are considered, the consistency loss can be proposed as:

[0086]

[0087] Here, β = 0.1 (or any other suitable value) is another boundary constant.

[0088] It is worth noting that the audio frame / signal pair {x i x j}1142 can be generated during the computation of pairwise sorting as shown above or in any other suitable manner. Furthermore, it can be generated, for example, from the pair x i and x j Extract the audio frame quadruple {x ik x il x jk x jl}1142 and uses a random small delay (e.g., less than 100ms) to generate the audio frame quadruple. As an example, and not a limitation, in order to generate the quadruple of the audio frame from a given pair {x} i x j Each quadruple can be formed as follows:

[0089] • Uniformly sample time delays between 0ms and 100ms. Use such a delay from x j Extract 1s frame x ik and x il and for x j x in jk and x jl Perform the same operation.

[0090] Similarly, the generated quadruple {x jk x il x jk x jl It can be stored together with degradation type and / or intensity information (e.g., stored as tag information).

[0091] Further information on possible methods for generating quadtuples can be found, for example, in section B.3 of the appendix.

[0092] Alternatively or alternatively, the pair {x} can also be obtained from the (predetermined) JND dataset 1143. i x j} and / or {x k x l}, then it can be obtained from those pairs {x i x j} and / or {x k x l Generate a quadruple {x} from} ik x il x jk xjl}

[0093] Same / Different Conditions

[0094] Using L as described above CONS The programmable data can also provide information about signal pairs corresponding to (essentially) the same degradation conditions, i.e., signals that have experienced the same type of degradation and (optionally) the same intensity. In other words, broadly speaking, the loss function can be considered to include a fourth loss function that indicates the degradation condition, and this fourth loss function can be calculated based on the difference between the label information, which includes relative degradation information, and its prediction.

[0095] In one possible example, this information can then be included by considering the classification loss in the first 1124.

[0096] L SD =BCE(δ) SD H SD (z u , z v (4)

[0097] Where BCE represents the binary cross-entropy, δ SD ∈{0,1} indicates the latent vector z u and z v Do they correspond to the same conditions? or Furthermore, H can be, for example, a small neural network 1132, which can take two vectors concatenated and produce a single probability value.

[0098] Minimum perceptible difference

[0099] As mentioned above, if signal pairs with artificial annotations regarding the perceived differences (or correlations) between signal pairs are accessible or available from the training set, this concept of perceived differences (or correlations) can be further reinforced in the latent space, for example, using another classification loss from Head 1125.

[0100] L JND =BCE(δ) JND H JND (z u , z v (5)

[0101] Where, δ JND ∈{0,1} indicates the latent representation z u and z v Whether it corresponds to JND. BCE (Binary Cross Entropy) and H (Small Neural Network 1133) can be the same as or similar to those shown above or take any other suitable form.

[0102] In other words, broadly speaking, the loss function can be considered to include a fifth loss function that indicates the JND index, and the fifth loss function can be calculated based on the difference between the label information, which includes the relative perceived difference, and its prediction.

[0103] Degeneration type

[0104] Another advantage of programmatically generated data is that if one starts with a signal considered pure or without significant degradation, it becomes clear which degradations have been applied. Therefore, in general, the degradation type head (sometimes also called the classification head) 1126 can acquire the latent vector z and further process it (e.g., via MLP 1134) to produce a probability output. It can then further use flags (e.g., label information) to calculate the binary cross-entropy, which signals the type of distortion in the original audio (if available). In other words, broadly speaking, the loss function can be considered to include a sixth loss function indicating the degradation type, and this sixth loss function can be calculated based on the difference between the label information, including information about the corresponding degradation function, and its prediction.

[0105] More specifically, in one possible implementation, the multi-class classification loss can be constructed as

[0106]

[0107] in, Indicator of latent representation z i Whether to include degenerate n. BCE (Binary Cross Entropy) and H (Neural Network 1134) can be the same as or similar to those shown above or take any other suitable form. In some examples, the case of no degeneration can also be included as one of the n possibilities, which is therefore considered to independently constitute a binary pure / degenerate classifier.

[0108] Degradation intensity

[0109] Generally, the degradation intensity head 1127 (sometimes also called the degradation head to distinguish it from the classification head 1126 shown above) can acquire the latent vector z and further process it (e.g., via MLP 1135) to produce an output, such as a value between 1 and 5. Then, if available (e.g., based on available label information), it can compute a regression-based loss using the degradation levels already introduced into the audio. In some implementations, these degradation levels can be recorded (stored) based on a (automatic) degradation algorithm applied before training the network / system. In other words, broadly speaking, the loss function can be considered to include a seventh loss function indicating the degradation intensity index, and the seventh loss function can be computed based on the difference between the label information, including the corresponding degradation intensity information, and its prediction.

[0110] More specifically (but not as a limitation), the intensity of the degradation can usually be determined (and applied) at the moment the degradation is applied to the signal. Therefore, in a possible example, the corresponding regressor can be added as...

[0111]

[0112] in, Indicates the intensity of degradation n.

[0113] Other quality assessment metrics

[0114] Finally, since the pair {x} has already been generated i x j Therefore, other or traditional reference-based (or no-reference) quality metrics can also be computed on these pairs and learned from them.

[0115] Generally, the regression head 1128 can acquire the latent vector z and further process it (e.g., via MLP 1136) to produce as many outputs as available or pre-computed alternative metrics for the audio under consideration (if available). In other words, broadly speaking, the loss function can be considered to include an eighth loss function indicating the regression metric, and the regression metric can be computed based on at least one of a reference-based quality metric and / or a referenceless quality metric.

[0116] In one possible implementation, the regression loss pool can be executed as follows:

[0117]

[0118] in, It is in {x i x j The value of the metric m calculated on}. In some examples, if needed, it can be based on the training data. Normalized to have zero mean and unit variance. Some possible examples of reference-based metrics may include (but are not limited to) perceptual evaluation of speech quality (PESQ), composite measurement of signal (CSIG), composite measurement of noise (CBAK), composite measurement of overall quality (COVL), segmented signal-to-noise ratio (SSNR), log-likelihood ratio (LLR), weighted slope spectral distance (WSSD), short-term objective intelligibility (STOI), scale-invariant signal-to-distortion ratio (SISDR), Mel-Cepstral distortion, and log-Mel band distortion. Of course, as those skilled in the art will recognize, any other suitable reference-based quality metric and / or reference-free quality metric may be used.

[0119] It is important to understand that each audio sample in the training set can be used for one or more (but not all) of the aforementioned loss functions. That is, some audio samples in the training set can be reused or shared by one or more loss functions. Figure 1B This is reflected and illustrated in the text. For example, the (algorithm-generated) audio sample 1142 used to calculate the loss function of the indicator score consistency head (index) 1123 can be reused when calculating the loss function of the indicator degradation condition head (index) 1124, and vice versa. This can significantly improve the efficiency of the training system. Furthermore, it should be noted that in some cases, it can be further configured to generate a final (overall) loss function for the training process based on one or more of a plurality of loss functions, for example, by utilizing an averaging process of those loss functions. As those skilled in the art will recognize, such a final loss function based on any number of suitable loss functions can be generated using any other suitable means or process, depending on various implementations and / or requirements.

[0120] Furthermore, it should be noted that the aforementioned heads 1121-1128 can be constructed from linear layers or MLPs (e.g., two-layer MLPs) with any suitable number of units (e.g., 400), and may also all have BNs at the ends. In some cases, it may be preferable to use simple heads to encourage the encoder, rather than the head, to learn high-level features that can be successfully utilized even by networks with limited capacity. In some cases, the decision on whether to use linear layers or MLPs may be based on the idea that the more relevant the auxiliary task is, the smaller the head capacity should be. Thus, in some implementations, linear layers can be empirically selected for the score s (i.e., 1131) and the JND and DT heads (i.e., 1133 and 1134, respectively). Note that setting linear layers for these three heads can provide interesting properties to the latent space, allowing it to reflect the 'distance' between latent representations (due to s and L). JND ), and promotes degenerate types of groups / clusters (due to L DT Of course, as technicians will recognize, any other suitable configuration can be applied here.

[0121] Figure 2 This is a flowchart illustrating an example of a method 200 for training a system based on deep learning (e.g., a neural network) to determine an indication of the audio quality of an audio input, according to an embodiment of the present disclosure. The system can, for example, be used with... Figure 1A The system 100 shown or as Figure 1B The system 110 shown is the same as or similar to the system shown.

[0122] Specifically, method 200 begins at step S210, obtaining at least one training set including audio samples as input. More specifically, the audio samples may include audio samples of a first type and audio samples of a second type, wherein each of the audio samples of the first type is labeled with information indicating a corresponding predetermined audio quality metric, and wherein each of the audio samples of the second type is labeled with information indicating a corresponding audio quality metric relative to a reference audio sample (e.g., relative to an audio quality metric of another audio sample in the training set). As indicated above, the reference audio sample used herein may be, but does not necessarily have to be, another audio sample in the training set. In other words, as those skilled in the art will understand and recognize, the reference audio sample may be an external reference audio sample (i.e., not in the training set) or an internal reference audio sample (i.e., in the training set).

[0123] As those skilled in the art will recognize, such a training set, comprising the desired audio samples (and appropriate labeling information), can be obtained (generated) in any suitable manner. For example, for the first type of audio samples, manually annotated audio data (samples, signals, frames) can be used, which can be obtained internally (e.g., by audio experts, ordinary listeners, or mechanics) or externally (e.g., using publicly available datasets). As an example, such manually annotated audio data could be MOS data, JND data, etc. Further information on possible datasets to be used for manual annotation can also be found, for example, in sections B.1 and B.2 of the appendix. On the other hand, for the second type of audio samples, programmatically generated audio data (samples, signals, frames) can be used, some examples of which have already been described above. Further information on possible datasets to be used for programmatic generation can also be found, for example, in section B.3 of the appendix.

[0124] Method 200 then proceeds to step S220, inputting the training set into a deep learning-based (neural network-based) system, such as... Figure 1A Input x 1000 or Figure 1B x 1100 in the middle.

[0125] Subsequently, method 200 executes step S230, iteratively training the system to predict the corresponding label information of audio samples in the training set. Specifically, training can be performed based on multiple loss functions, and multiple loss functions can be generated to reflect the difference between the label information of the audio samples in the training set and their corresponding predictions, as referenced above. Figure 1B As shown in the diagram.

[0126] Generally, the entire network / system can be trained end-to-end, for example using stochastic gradient descent and backpropagation. Before training, a set of audio samples can be obtained as described above, and several degradations can be performed on them. As those skilled in the art will recognize, various suitable degradations applied to it can include, but are not limited to, operations / processes involving reverberation, clipping, encoding it with different codecs, phase distortion, inverting it, adding (real or artificial) background noise, etc. Some possible degradations are given below as examples, but not as limitations:

[0127] • Add real noise (from different sources).

[0128] • Added artificial noise (generated colored noise).

[0129] • Added tone / humming sound.

[0130] • Audio resampling.

[0131] μ-law quantization.

[0132] • Wave clipping.

[0133] • Audio inverted.

[0134] • Insert mute.

[0135] Insertion noise.

[0136] Insertion attenuation.

[0137] • Disturbance amplitude.

[0138] ·Delay.

[0139] Equalized, bandpass, and bandstop filtering.

[0140] • Low / high-pass filtering.

[0141] ·chorus.

[0142] • Overload.

[0143] • Phase shifter.

[0144] • Pitch shifting.

[0145] ·reverberation.

[0146] ·vibrato.

[0147] • Phase distortion: Griffin-Lim, random phase, shuffled phase, spectral holes, spectral convolution.

[0148] • Transcoding (encoding and decoding with an audio codec and then re-encoding and decoding).

[0149] It is worth noting that degradation can be applied in a non-fixed manner to the entire audio frame or only to a portion thereof. Furthermore, in some cases, existing (automatic) measurements can be run on these audio pairs. The primary use of automatically generated data is to supplement manually annotated data; however, one can still train publicly available networks or systems without either and still obtain reasonable results with minimal adaptation.

[0150] Further information about possible degradation functions and their corresponding degradation intensities can also be found, for example, in Part C of the Appendix.

[0151] The system can be trained in any suitable manner based on any appropriate configuration or settings. For example, in some possible implementations, the system can be trained using the RangerQH optimizer, for instance, by using default parameters and 10. -3 The learning rate is used for training. The learning rate may decay by a factor (e.g., 1 / 5 at 70% and 90% training). Furthermore, to facilitate generalization and slightly improve performance, a randomized weighted average may be used in the final training epoch if needed. Since all losses are typically likely to be in similar ranges after several iterations, loss weighting may not be performed.

[0152] Once training is complete, the trained system can then be used or manipulated to determine quality indicators of the input audio. (See now for reference.) Figure 3 The diagram illustrates a flowchart of an example of a method 300 for training a system based on deep learning (e.g., a neural network) to determine an indication of the audio quality of an audio input, according to an embodiment of the present disclosure. The system can, for example, be used with... Figure 1A The system 100 shown or as Figure 1B The system 110 shown is the same as or similar to the system shown. That is, the system may include a suitable coding level and a suitable evaluation level, as shown in any of the figures. Furthermore, the system may have undergone, for example... Figure 2 The training process is illustrated in the diagram. Therefore, for the sake of brevity, a repeated description of it can be omitted.

[0153] Specifically, method 300 may begin at step S310, where the input audio samples are mapped to a feature space representation (e.g., a latent space representation z as shown above) by the coding level.

[0154] Then, method 300 can proceed to step S320, whereby the evaluation level predicts information indicating a predetermined audio quality metric and information indicating a relative audio quality metric relative to a reference audio sample based on the feature space representation. The predicted information (e.g., information indicating a relative audio quality metric relative to a reference audio sample) can be used to further train (adjust) the system, as described above. Figure 1B Detailed explanation.

[0155] In this way, scores can be generated (e.g., such as...). Figure 1B The final quality metric, such as the score (s1140), is such that the output metric (or score) can then be used as an indication of the quality of the input audio samples. As mentioned above, the metric (or score) can be generated as any suitable representation, such as a value between 1 and 5 (e.g., 1 or 5 indicates the highest audio quality).

[0156] In summary, broadly speaking, this disclosure proposes a semi-supervised approach to learn a speech quality model that incorporates multiple objectives. In some cases, the disclosed method may also be simply referred to as semi-supervised speech quality assessment (or SESQA). Specifically, this disclosure learns from existing labeled data as well as (theoretically unlimited) amounts of unlabeled or programmably generated data, producing speech quality scores along with available latent features and information-assisted outputs. In a multi-task setting, the scores and outputs can be simultaneously optimized using several different but complementary objective criteria, as relevant cues exist within all of these criteria. By allowing information to flow through shared latent spatial bottlenecks, the considered objectives learn to cooperate and promote better and more robust representations while discarding unnecessary information.

[0157] It is worth noting that this disclosure can be utilized in a variety of ways, such as (but not limited to):

[0158] • As a cloud API used to obtain quality scores for uploaded audio.

[0159] • As a tool for monitoring communications.

[0160] • As a tool for monitoring codec degradation.

[0161] • As a tool for evaluating the performance of audio processing algorithms (e.g., internally).

[0162] • As a loss function used to train or tune deep learning models (e.g., neural network models).

[0163] • As a feature extractor used to identify the type of distortion present in an audio signal.

[0164] Of course, as the technicians will understand and recognize, any other suitable use case may be utilized.

[0165] Figure 4 Figures 8 through 8 are example diagrams illustrating various results and comparisons based on the embodiments(s) of this disclosure. In particular, quantitative comparisons are performed using many existing or conventional methods. Details relating to some of the existing methods used for comparison can be found, for example, in Part D of the Appendix.

[0166] Furthermore, it should be noted that for evaluation purposes, this disclosure typically uses three MOS datasets: two internal datasets and one publicly available dataset. The first internal dataset consists of 1,109 recordings and a total of 1.5 hours of audio, characterized primarily by user-generated content (UGC). The second internal dataset consists of 8,016 recordings and 15 hours of audio, characterized by telephone and VoIP degradation. The third dataset is TCD-VoIP, which consists of 384 recordings and 0.7 hours of audio, characterized by significant VoIP degradation. Another dataset we use is the JND dataset, which consists of 20,797 pairs of recordings and 28 hours of audio. More details about the training sets can be found, for example, in Part B of the Appendix. For the programmatic generation of the data, this disclosure typically uses one set of internal datasets and the public dataset, and generates 70,000 quadruplets corresponding to 78 hours of audio. Furthermore, a total of 37 possible degradations were employed, including additional background noise, hum, clipping, sound effects, packet loss, phase distortion, and various audio codecs (more details can be found in Part C of Appendix). This disclosure is then compared with ITU-P563, both methods being feature-based losses, one using JND (FL-JND) and the other using PASE (FL-PASE), SRMR, Auto-MOS, Quality Network, WEnet, CNN-ELM, and NISQA. For evaluation purposes, some of these methods have been reimplemented to fit the training and evaluation pipeline of this disclosure, if necessary / possibly, and have been adapted to operate at 48kHz. Note that FL, AutoMOS, and NISQA typically utilize additional data outside of MOS, and are therefore weakly semi-supervised methods. More details about the baseline methods can also be found in Part D of Appendix.

[0167] All methods were placed under the same settings to select their optimal optimizer and hyperparameters on the validation set. Data was trained for 5 epochs using 1-second weakly labeled frames by performing data augmentation and reusing MOS data (e.g., an epoch can be defined as a complete pass over programmatically generated data). Random scaling, phase reversal, and temporal sampling were also used as data augmentation. For evaluation, L... MOS and L CONS And calculate the misclassification rate R. RANK (Regarding the interpretability report R) RANK Instead of L RANK Additionally, calculate the summation error E. TOTAL =0.5L MOS +RR ANK +L CONS(A weight of 0.5 is introduced to compensate for different ranges). Five-fold cross-validation is also performed and the average error is reported.

[0168] Of course, it should be understood that any other suitable training dataset and / or evaluation method may be used, depending on the implementation method and / or requirements.

[0169] Based on these results, the method disclosed in this disclosure appears to outperform methods among the previously considered evaluation metrics. It was also observed that the scores obtained from the scorehead correlate well with human judgments of quality, are able to detect a wide range of varying degrees of degradation with different levels of distortion, and that the latent space z clusters degradation types.

[0170] For example, Figure 4 Scores typically appear to correlate well with human judgment.

[0171] Figure 5 The empirical distribution of distances between latent space vectors z is shown. Figure 510 shows that smaller distances correspond to similar utterances with the same degradation type and intensity (e.g., mean distance 7.6 and standard deviation 3.4), while Figure 530 shows that larger distances correspond to different utterances with different degradations (e.g., mean distance 16.9 and standard deviation 3.9). The overlap between these two distributions appears to be small, with the mean plus one standard deviation being non-intersecting. Similar utterances with different degradations (Figure 520) extend between the first two distributions (e.g., mean distance 13.7 and standard deviation 5.5). This is significant in a latent space organized by degradation and intensity, with a large range between smaller and larger intensities. It can be assumed that this overall behavior is a consequence of all losses, but specifically s and L. JND The result of its (linear) head.

[0172] Figure 6A The study depicts how the score *s* calculated from a test signal without degradation tends to decrease with increasing degradation intensity. In many cases, the effect appears both clearly visible and consistent (e.g., added noise or EAC3 codec). In other cases, the effect of high intensity appears to saturate (e.g., μ-law quantization or clipping). There are also several types of degradation where the intensity does not correspond to a single variable and therefore the effect appears less noticeable. Overall, consistent behavior between degradation and intensity is observed. It can be assumed that L... MOS L RANK and L DS This is likely the main driving force behind achieving this behavior. Figure 6B and Figure 6C Similar additional results are illustrated schematically, where the scores appear to reflect the progressive audio degradation quite well.

[0173] Figure 7AThree low-dimensional t-SNE projections of the latent space vector z are shown. The figure illustrates how different degradation types group or cluster together. For example, in the case of complexity 200, the latent vectors of frames containing added noise are clustered at the center. Interestingly, similar degradations can also be seen to be placed close to each other. This is the case for added noise and colored noise, MP3 and OPUS codecs, or Griffin-Lim and STFT phase distortion. It can be hypothesized that this clustering behavior might be L DT The direct result of its (linear) head.

[0174] Figure 7B Similar additional results are illustrated schematically, in which the classification head appears to have the potential to distinguish between degeneration types.

[0175] Figure 8A This schematically illustrates a comparison with some existing or traditional methods. From Figure 8A Overall, all methods appear to significantly outperform the randomized baseline, and about half of these methods seem to achieve an error comparable to that of human scores (estimated by taking the standard deviation across listeners and averaging it across discourses). MOS It was also observed that many existing methods reported fairly good consistency, among which L CONS Within the range of 0.1, it is six times lower than the random baseline. However, when considering relative pairwise ordination (R0... RANK When considering existing methods, considerable errors are introduced. This disclosure appears to significantly outperform all listed existing methods across all evaluation metrics considered, including standard L. MOS The only exception previously stated appears to be L in the ITU-P563 method. CONS The metric, however, appears to have a high L CONS And almost random R RANK Considering the aggregate indicator E TOTAL This disclosure appears to reduce the error of the best existing method by 36%.

[0176] Figure 8B The impact of the considered criteria / tasks on the performance of the disclosed method is illustrated schematically. First, it is observed that removing a single criterion never seems to reduce the error. This may suggest that, in terms of performance, these errors appear to be harmless. Next, it is observed that there are some relevant criteria (e.g., L) that have a considerable impact when removed. MOS and L RANK However, the lack of one of these relevant criteria does not necessarily result in the average error of existing methods (see, for example...). Figure 8A E in TOTALRegarding some less relevant tasks, note that they still seem useful for the outputs they produce (e.g., knowing whether a pair of signals exhibits a JND difference) or for the properties they impart to the organization of the latent space z. Finally, it is also interesting to emphasize considering L in isolation. MOS Standard (see Figure 8B The last line appears to produce performance comparable to some of the best-performing existing methods (see, for example...). Figure 8A (NISQA and CNN-ELM in [reference]). In summary, this demonstrates that considering multiple optimization criteria and tasks appears to be key to achieving excellent performance, and empirically demonstrates a semi-supervised approach to audio quality evaluation similar to that disclosed herein.

[0177] Figure 8C The results of further evaluating the generalization ability of the considered method by performing ex post-hoc informal testing on out-of-sample data are illustrated. For this purpose, for example, 20 new recordings can be selected from UGC characterized by clean or production-quality speech, as well as speech with degradations such as real background noise, codec artifacts, or microphone distortion. A new group of listeners might then be asked to rate the quality of the recordings on a scale of 1 to 5, and their ratings compared to those generated by a model pre-trained on an internal UGC dataset. Figure 8C As can be seen, the ranking of existing methods has changed, showing that some methods are better than others in generalizing out-of-sample data. Nevertheless, this disclosure still appears to outperform by a large margin across all listed metrics. It is noteworthy that the L... (The sentence is incomplete and requires further context to translate accurately.) MOS This represents a 21% reduction, compared to the 7% observed in the in-sample data (from...). Figure 8A The relative L MOS The differences are much larger. This may indicate that this disclosure generalizes better to out-of-sample but relevant data.

[0178] Figure 8D and Figure 8E The error values ​​for the considered dataset and the L values ​​across the dataset are further illustrated schematically. TOTAL Average value. Specifically, Figure 8D This disclosure is illustrated in comparison with existing methods, and Figure 8E This schematically illustrates that, in addition to using only L, MOS The training effect is not achieved by using one of the losses considered. It is worth noting that, similar to what was mentioned above, E TOTAL =0.5L MOS +R RANK +L CONS . Figure 8FFurther additional results are provided, which schematically demonstrate that the method presented in this disclosure (last line) appears to be superior to the conventional methods listed.

[0179] The foregoing has described possible methods for training and operating a deep learning-based (e.g., neural network-based) system for determining an indication of the audio quality of an input audio sample, as well as possible implementations of such a system. Additionally, this disclosure relates to apparatus for implementing these methods. Examples of such apparatus may include a processor (e.g., a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), one or more application-specific integrated circuits (ASICs), one or more radio frequency integrated circuits (RFICs), or any combination thereof) and memory coupled to the processor. The processor may be adapted to perform some or all of the steps of the methods described throughout the disclosure.

[0180] The apparatus may be a server computer, client computer, personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), cellular phone, smartphone, web device, network router, switch or bridge, or any machine capable of (sequentially or otherwise) executing instructions specifying actions to be taken by said apparatus. Furthermore, this disclosure will relate to any collection of apparatuses that individually or in combination execute instructions to perform any one or more methods discussed herein.

[0181] This disclosure further relates to a program (e.g., a computer program) that includes instructions that, when executed by a processor, cause the processor to perform some or all of the steps of the methods described herein.

[0182] Furthermore, this disclosure relates to computer-readable (or machine-readable) storage media for storing the aforementioned programs. Here, the term "computer-readable storage medium" includes, but is not limited to, data storage libraries in the form of, for example, solid-state memory, optical media, and magnetic media.

[0183] Unless otherwise specifically stated, it is obvious from the following discussion that, throughout this public discussion, terms such as “processing,” “computing,” “calculating,” “determining,” and “analyzing” are used to refer to the actions and / or processes by which data represented as physical (e.g., electronic) quantities are manipulated and / or transformed into other data similarly represented as physical quantities by a computer or computing system or similar electronic computing device.

[0184] In a similar manner, the term "processor" can refer to any device or part of a device that processes electronic data, for example, from registers and / or memory, to transform that electronic data into other electronic data, for example, that can be stored in registers and / or memory. "Computer," "computing machine," or "computing platform" can include one or more processors.

[0185] In one example embodiment, the methods described herein can be executed by one or more processors that accept computer-readable (also known as machine-readable) code containing a set of instructions that, when executed by the one or more processors, perform at least one of the methods described herein. This includes any processor capable of executing a set of instructions (sequential or otherwise) specifying an action to be taken. Thus, an example is a typical processing system comprising one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem comprising main RAM and / or static RAM and / or ROM. A bus subsystem may be included for communication between components. The processing system may further be a distributed processing system in which processors are coupled together via a network. If the processing system requires a display, it may include such a display, for example, a liquid crystal display (LCD) or a cathode ray tube (CRT) display. If manual data input is required, the processing system also includes one or more input devices, such as alphanumeric input units (e.g., a keyboard), pointing control devices (e.g., a mouse), etc. The processing system may also encompass storage systems such as disk drive units. In some configurations, the processing system may include sound output devices and network interface devices. The memory subsystem therefore includes a computer-readable carrier medium carrying computer-readable code (e.g., software) comprising a set of instructions that, when executed by one or more processors, cause one or more of the methods described herein to be performed. It should be noted that when the methods comprise several elements (e.g., several steps), no particular order of these elements is implied unless specifically stated otherwise. During the execution of software by a computer system, the software may reside on a hard disk, or it may reside wholly or at least partially in RAM and / or a processor. Therefore, the memory and processor also constitute a computer-readable carrier medium carrying computer-readable code. Furthermore, the computer-readable carrier medium may be formed or included in a computer program product.

[0186] In alternative example embodiments, one or more processors may operate as standalone devices or may be connected to (e.g., networked to) other processors in a networked deployment. These processors may operate as server or user machines in a server-user network environment, or as peer-to-peer machines in a peer-to-peer or distributed network environment. The one or more processors may form a personal computer (PC), tablet PC, personal digital assistant (PDA), cellular phone, web facility, network router, switch, or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) specifying the actions to be taken by that machine.

[0187] It should be noted that the term "machine" should also be considered to include any set of machines that individually or jointly execute a set (or more sets) of instructions to perform any or more of the methods discussed herein.

[0188] Therefore, an example embodiment of each method described herein takes the form of a computer-readable carrier medium carrying a set of instructions, such as a computer program for execution on one or more processors (e.g., one or more processors as part of a web server arrangement). Thus, as those skilled in the art will recognize, example embodiments of this disclosure can be embodied as methods, apparatus such as dedicated devices, apparatus such as data processing systems, or computer-readable carrier media (e.g., computer program products). A computer-readable carrier medium carries computer-readable code comprising a set of instructions that, when executed on one or more processors, cause one or more processors to implement the method. Therefore, aspects of this disclosure can take the form of methods, entirely hardware example embodiments, entirely software example embodiments, or example embodiments combining software and hardware aspects. Furthermore, this disclosure can take the form of a carrier medium (e.g., a computer program product on a computer-readable storage medium) carrying computer-readable program code embodied in the medium.

[0189] Software can be further sent or received over a network via a network interface device. While the carrier medium is a single medium in the example embodiment, the term "carrier medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store one or more sets of instructions. The term "carrier medium" should also be considered to include any medium capable of storing, encoding, or carrying a set of instructions for execution by one or more processors and causing one or more processors to perform any one or more methods of this disclosure. The carrier medium can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical discs, magnetic disks, and magneto-optical discs. Volatile media include dynamic memory, such as main memory. Transmission media include coaxial cables, copper wires, and optical fibers, including conductors containing a bus subsystem. Transmission media can also take the form of acoustic or optical waves, such as those generated during radio wave and infrared data communication. For example, the term "carrier medium" should therefore be considered to include, but is not limited to, solid-state storage; computer products embodied in optical and magnetic media; media carrying propagation signals that can be detected by at least one or more processors and represent a set of instructions, which, when executed, implement a method; and transmission media in a network that carry propagation signals that can be detected by at least one of one or more processors and represent the set of instructions.

[0190] It will be understood that, in one exemplary embodiment, the steps of the method under discussion are performed by a suitable processor (or processors) in a processing (e.g., a computer) system that executes instructions (computer-readable code) stored in a storage device. It will also be understood that this disclosure is not limited to any particular implementation or programming technique, and that this disclosure can be implemented using any suitable technique for implementing the functions described herein. This disclosure is not limited to any particular programming language or operating system.

[0191] Throughout this disclosure, references to “one example embodiment,” “some example embodiments,” or “example embodiment” mean that a particular feature, structure, or characteristic described in connection with an example embodiment is included in at least one example embodiment of this disclosure. Therefore, the phrases “in one example embodiment,” “in some example embodiments,” or “in an example embodiment” appearing throughout this disclosure do not necessarily refer to the same example embodiment. Furthermore, in one or more example embodiments, particular features, structures, or characteristics may be combined in any suitable manner, as will be apparent to those skilled in the art based on this disclosure.

[0192] As used herein, unless otherwise specified, ordinal adjectives such as “first,” “second,” “third,” etc., are used to describe common objects only to indicate different instances of similar objects and are not intended to imply that the objects described must be in a given order in time, space, hierarchy, or any other way.

[0193] In the claims below and in the description herein, the terms *comprising*, *comprised of*, or *which comprises* are open-ended terms meaning that at least the following element / feature is included, but not excluding other elements / features. Therefore, when the term *comprising* is used in a claim, it should not be construed as limited to the means, elements, or steps listed thereafter. For example, the expression of a device including A and B should not be limited to a device that includes only elements A and B. As used herein, the terms *including*, *which includes*, or *that includes* are also open-ended terms meaning that at least the element / feature following the term is included, but not excluding other elements / features. Therefore, *including* is synonymous with *comprising* and means *comprising*.

[0194] It should be recognized that in the foregoing description of exemplary embodiments of this disclosure, various features of this disclosure are sometimes combined in a single exemplary embodiment / figure or its description in order to simplify the disclosure and aid in understanding one or more of the inventive aspects. However, the approach of this disclosure should not be construed as reflecting an intention in the claims to require more features than expressly recited in each claim. Rather, as reflected in the following claims, the inventive aspects lie in fewer than all features of a single foregoingly disclosed exemplary embodiment. Therefore, the claims following this specification are hereby expressly incorporated, wherein each claim is an independent, separate exemplary embodiment of this disclosure.

[0195] Furthermore, while some of the exemplary embodiments described herein include features that are included in other exemplary embodiments but not others, as those skilled in the art will understand, combinations of features from different exemplary embodiments are intended to be within the scope of this disclosure and to form different exemplary embodiments. For example, any exemplary embodiment of the claimed embodiments in the appended claims can be used in any combination.

[0196] Numerous specific details are set forth in the description provided herein. However, it should be understood that exemplary embodiments of this disclosure may be practiced without these specific details. In other instances, well-known methods, structures, and techniques have not been shown in detail to avoid obscuring the understanding of this specification.

[0197] Therefore, although the mode considered to be the best mode of this disclosure has been described, those skilled in the art will recognize that other and further modifications can be made thereto without departing from the spirit of this disclosure, and all such changes and modifications falling within the scope of this disclosure are intended to be claimed. For example, any formulas given above merely represent processes that can be used. Functions can be added or removed from the block diagram, and operations can be interchanged between functional blocks. Steps can be added or removed from the methods described within the scope of this disclosure.

[0198] The foregoing has described exemplary embodiments (“EEE”) of this disclosure with respect to methods and systems for determining an indication of the audio quality of an audio input. Therefore, embodiments of the invention may relate to one or more of the following enumerated examples:

[0199] EEE 1. A method for training a convolutional neural network (CNN) to determine an audio quality rating of an audio signal, the method comprising:

[0200] The audio signal is transformed into a low-dimensional latent space representation of the audio signal;

[0201] The low-dimensional latent space representation of the audio signal is input into the encoder stage;

[0202] The low-dimensional latent space representation audio signal is processed via the encoder level to determine the parameters of the low-dimensional latent space representation audio signal;

[0203] The audio quality score of the audio signal is determined based on the parameters and the low-dimensional latent space representation of the audio signal.

[0204] EEE 2. A method for training a deep learning-based system to determine an indication of the audio quality of an audio input, the method comprising:

[0205] At least one training set including audio samples is obtained as input, wherein the audio samples include audio samples of a first type and audio samples of a second type, wherein each of the audio samples of the first type is labeled with information indicating a corresponding predetermined audio quality metric, and wherein each of the audio samples of the second type is labeled with information indicating a corresponding audio quality metric relative to an audio quality metric of a reference audio sample or an audio quality metric of another audio sample in the training set.

[0206] The training set is input into the deep learning-based system; and

[0207] The system is iteratively trained to predict the corresponding label information of the audio samples in the training set, wherein the training is based on multiple loss functions; and

[0208] The multiple loss functions are generated to reflect the difference between the label information of the audio samples in the training set and their corresponding predictions.

[0209] EEE 3. The method according to EEE 2, wherein the first type of audio samples includes manually annotated audio samples, each manually annotated audio sample being labeled with information indicating the corresponding predetermined audio quality index.

[0210] EEE 4. The method according to EEE 3, wherein the manually annotated audio samples include Mean Opinion Score (MOS) audio samples and / or Minimum Perceptible Difference (JND) audio samples.

[0211] EEE 5. The method according to any one of the preceding EEEs, wherein the second type of audio samples comprises algorithmically generated audio samples, each algorithmically generated audio sample being labeled with information indicating a relative audio quality metric.

[0212] EEE 6. The method according to EEE 5, wherein each of the samples generated by the algorithm is generated by selectively applying at least one degradation function, each having a corresponding degradation intensity, to a reference audio sample or an audio sample generated by another algorithm, and wherein the tag information includes information indicating the corresponding degradation function and / or corresponding degradation intensity that has been applied to the reference audio sample or the audio sample generated by the other algorithm.

[0213] EEE 7. The method according to EEE 6, wherein the label information further includes information indicating degradation relative to the reference audio sample or other audio samples in the training set.

[0214] EEE 8. The method according to EEE 6 or 7, wherein the degradation function is selected from a plurality of available degradation functions, and / or wherein the corresponding degradation intensity is set such that the degradation is perceptibly apparent at its minimum value.

[0215] EEE 9. The method according to EEE 8, wherein the plurality of available degradation functions include functions associated with one or more of the following: reverberation, clipping, encoding with different codecs, phase distortion, audio inversion, and background noise.

[0216] EEE 10. The method according to any one of EEE 6 to 9, wherein the audio samples generated by the algorithm are generated as audio frame pairs {x} i x j} and / or audio frame quadruple Among them, audio frame x i It is generated by selectively applying at least one degradation function, each with a corresponding degradation intensity, to a reference audio frame, where the audio frame x j This is achieved by selectively applying at least one degradation function, each with a corresponding degradation intensity, to the audio frame x. i To generate, among which, audio frames and This is achieved by selectively applying a corresponding time delay to audio frame x. i From the audio frame x i Extracted from, and among them, audio frames and This is achieved by selectively applying a corresponding time delay to audio frame x. j From the audio frame x j Extracted from.

[0217] EEE 11. The method according to any one of the preceding EEEs, wherein the loss function includes a first loss function indicating a MOS error metric, and wherein the first loss function is calculated based on the difference between the true MOS data of the audio samples in the training set and the predictions for the audio samples.

[0218] EEE 12. The method according to any one of EEE 5 to 10 or EEE 11 when belonging to any one of EEE 5 to 10, wherein the label information of the second type of audio samples includes relative information indicating whether one audio sample degrades more than another audio sample, wherein the loss function includes a second loss function indicating a pairwise ranking metric, and wherein the second loss function is calculated based on the ranking established by the label information including the relative degradation information and its predictions.

[0219] EEE 13. The method according to EEE 12, wherein the system is trained in such a way that an audio sample with less degradation obtains an audio quality index indicating better audio quality than another audio sample with more degradation.

[0220] EEE 14. The method according to any one of EEE 5 to 10, 12 and 13 or EEE 11 when belonging to any one of EEE 5 to 10, wherein the label information of the second type of audio samples includes relative information indicating perceptual relevance between audio samples, wherein the loss function includes a third loss function indicating a consistency index, and wherein the third loss function is calculated based on the difference between the label information including the perceptual relevance information and its prediction.

[0221] EEE 15. The method according to EEE 14, wherein the consistency index indicates whether two or more audio samples have the same degradation function and degradation intensity and whether they correspond to the same time frame.

[0222] EEE 16. The method according to any one of EEE 5 to 10 and 12 to 15 or EEE 11 when belonging to any one of EEE 5 to 10, wherein the label information of the second type of audio sample includes relative information indicating whether an audio sample has been applied with the same degradation function and the same degradation intensity as another audio sample, wherein the loss function includes a fourth loss function indicating a degradation condition index, and wherein the fourth loss function is calculated based on the difference between the label information including the relative degradation information and its prediction.

[0223] EEE 17. The method according to any one of EEE 5 to 10 and 12 to 16 or EEE 11 when belonging to any one of EEE 5 to 10, wherein the label information of the second type of audio samples includes relative information indicating perceptual differences relative to each other, wherein the loss function includes a fifth loss function indicating a JND index, and wherein the fifth loss function is calculated based on the difference between the label information including relative perceptual differences and its prediction.

[0224] EEE 18. The method according to any one of EEE 5 to 10 and 12 to 17 or EEE 11 when belonging to any one of EEE 5 to 10, wherein the label information of the second type of audio sample includes information indicating a degradation function that has been applied to the audio sample, wherein the loss function includes a sixth loss function indicating a degradation type index, and wherein the sixth loss function is calculated based on the difference between the label information including the corresponding degradation function information and its prediction.

[0225] EEE 19. The method according to any one of EEE 5 to 10 and 12 to 18 or EEE 11 when belonging to any one of EEE 5 to 10, wherein the label information of the second type of audio sample includes information indicating the degradation intensity that has been applied to the audio sample, wherein the loss function includes a seventh loss function indicating a degradation intensity index, and wherein the seventh loss function is calculated based on the difference between the label information including the corresponding degradation intensity information and its prediction.

[0226] EEE 20. The method according to any one of the preceding EEEs, wherein the loss function includes an eighth loss function indicating a regression index, and wherein the regression index is calculated based on at least one of a reference-based quality metric and / or a referenceless quality metric.

[0227] EEE 21. The method according to EEE 20, wherein the reference-based quality metric includes at least one of the following: PESQ, CSIG, CBAK, COVL, SSNR, LLR, WSSD, STOI, SISDR, Mel cepstral distortion, and log-Mel band distortion.

[0228] EEE 22. The method according to any one of the preceding EEEs, wherein each of the audio samples in the training set is used for at least one of the plurality of loss functions, and wherein the final loss function used for the training is generated based on an averaging process of one or more of the plurality of loss functions.

[0229] EEE 23. The method according to any one of the preceding EEEs, wherein the system includes an encoding level for mapping the audio input to a feature space representation and an evaluation level for generating a prediction of label information based on the feature space representation.

[0230] EEE 24. The method according to any one of the preceding EEEs, wherein the encoding level for generating the intermediate representation includes a neural network encoder.

[0231] EEE 25. The method according to any one of the preceding EEEs, wherein each of the plurality of loss functions is determined based on a neural network comprising a linear layer or a multilayer perceptron (MLP).

[0232] EEE 26. A deep learning-based system for determining an indication of the audio quality of an input audio sample, wherein the system comprises:

[0233] Coding level; and

[0234] Assessment level,

[0235] The coding level is configured to map the input audio samples to a feature space representation; and

[0236] The evaluation level is configured to predict information indicating a predetermined audio quality metric based on the feature space representation, and further predict information indicating a relative audio quality metric relative to another audio sample.

[0237] EEE 27. The system according to EEE 26, wherein the system is configured to:

[0238] At least one training set is used as input, wherein the training set includes audio samples of a first type and audio samples of a second type, wherein each of the audio samples of the first type is labeled with information indicating a corresponding predetermined audio quality metric, and wherein each of the audio samples of the second type is labeled with information indicating a corresponding audio quality metric relative to a reference audio sample or relative to an audio quality metric of another audio sample in the training set.

[0239] Input the training set into the system; and

[0240] The system is iteratively trained based on the training set to predict the corresponding label information of the audio samples in the training set based on multiple loss functions, which are generated to reflect the difference between the label information of the audio samples in the training set and their corresponding predictions.

[0241] EEE 28. A method for operating a neural deep learning-based system for determining an indication of the audio quality of an input audio sample, wherein the system includes an encoding level and an evaluation level, the method comprising:

[0242] The input audio samples are mapped to a feature space representation by the coding level; and

[0243] The evaluation level predicts information indicating a predetermined audio quality metric and information indicating a relative audio quality metric relative to another audio sample, based on the feature space representation.

[0244] EEE 29. A program comprising instructions that, when executed by a processor, cause the processor to perform the steps of the method according to any one of EEE 1 to 25 and 28.

[0245] EEE 30. A computer-readable storage medium storing a program according to EEE 29.

[0246] EEE 31. An apparatus comprising a processor and a memory coupled to the processor, wherein the processor is adapted to cause the apparatus to perform the steps of the method according to any one of EEE 1 to 25 and 28.

[0247] appendix

[0248] Appendix A. Calculating the score using the reference signal

[0249] In order to calculate the score s with a reference-based setting rather than a reference-free setting, two signals x i and x j The corresponding potential value z is obtained through the encoder. i and z j Then, for example, using the linear unit of these two potential values ​​to calculate s. ij =1+4σ(w T z i -w T z j +b). Other options calculate a single score s based on the latent vector difference. ij =1+4σ(w T (z i -z j )+b), or concatenate the latent values ​​and use layers of twice the size. Other perspectives include replacing vector differences or linear layers with more complex nonlinear, parameterized, and / or learnable functions.

[0250] Appendix B: Data

[0251] As mentioned, three (3) data types were used in the semi-supervised method: MOS data, JND data, and program-generated data. Additional out-of-sample datasets used in the post-hoc hearing tests are summarized in the description, and their degradation characteristics are similar to those in the internal UGC dataset (see below).

[0252] B.1. MOS Data

[0253] The entire network / system was trained and evaluated on three (3) different MOS datasets with different sizes and characteristics:

[0254] 1. Internal UGC Dataset – This dataset consists of 1,109 UGC recordings, totaling 1.5 hours of audio. All recordings were converted to 48kHz mono WAV PCM and normalized to have the same loudness. Speech ranges from single words to several sentences, spoken by male and female speakers in various conditions using different languages ​​(primarily English, but also including Chinese, Russian, Spanish, etc.). Common degradations in the recordings include background noise (street, cafeteria, wind, background TV / radio, other people's speech, etc.), reverberation, bandwidth reduction (low-pass down to 3kHz), and encoding artifacts (MP3, OGG, AAC, etc.). Quality ratings were collected with the help of a group of 10 expert listeners with at least several years of experience in audio processing / engineering. The recordings have ratings between 4 and 10, which are obtained by following standard procedures described, for example, by IEEE and ITU (see PCLoizou, “Speech quality assessment”, Multimedia Analysis, Processing and Communications, Studies in Computational Intelligence, Berlin: Springer, 2011, Vol. 346, pp. 623-654, and references therein).

[0255] 2. Internal Telephone / VoIP Dataset – This dataset consists of 8,016 recordings with typical telephone and VoIP degradation, totaling 15 hours of audio. With the exception of a small portion, all audio was initially recorded at 48kHz and normalized to have the same loudness before further processing. The recordings contain two sentences separated by silence and range in duration from 5 to 15 seconds, following a protocol similar to ITU-P800. Male and female speech are balanced, and different languages ​​are present (English, French, Italian, Czech, etc.). Common degradations include packet loss (between 20ms and 60ms), bandwidth reduction (low-pass down to 3kHz), added synthetic noise (different SNRs), and coding artifacts (G772, OPUS, AC3, etc.). Quality ratings were provided by a fixed group of listeners, with each recording receiving between 10 and 15 ratings. The rating is obtained by following the standard procedures described by the ITU (see PCLoizou, “Voice Quality Assessment”, Multimedia Analysis, Processing and Communications, Computational Intelligence Research, Berlin: Springer, 2011, Vol. 346, pp. 623-654, and references therein).

[0256] 3. TCD-VoIP Dataset – This is a public dataset available online at http: / / www.mee.tcd.ie / ~sigmedia / Resources / TCD-VoIP. The dataset consists of 384 recordings with common VoIP degradation, totaling 0.7 hours. A good description of the dataset is provided in the original reference (N. Harte, E. Gillen, and A. Hines, “TCD-VoIP, a research database of degraded speech for assessing quality in VoIP applications”, Proc. of the Int. Workshop on Quality of Multimedia Experience (QoMEX), 2015). Although it also features VoIP degradation, some of it differs from our intranet / VoIP dataset (in both type and intensity).

[0257] B.2. JND Data

[0258] JND data was also used for training. A dataset compiled by Manocha et al. (P. Manocha, A. Finkelstein, Z. Jin, NJ Bryan, R. Zhang, and GJ Mysore, “A differentiable perceptual audio metric learned from just noticeable differences”, ArXiv:2001.04460, 2020) was used, which is available at https: / / github:com / pranaymanocha / PerceptualAudio. The dataset consists of 20,797 pairs of “disturbed” recordings (28 hours of audio), each pair from the same utterance, with annotations indicating whether the disturbance is visible in pairs. The annotations were crowdsourced from Amazon Mechanical Turk according to a specific procedure (P. Manocha, A. Finkelstein, Z. Jin, NJ Bryan, R. Zhang, and GJ Mysore, "Discriminative Perceptual Audio Metrics Learned from Minimal Perceptible Difference", ArXiv:2001.04460, 2020). The perturbations correspond to additional linear background noise, reverberation, and encoding / compression.

[0259] B.3. Data generated by programming

[0260] Quadruple {x ik x il x jk x jl The results were calculated based on programmatically generated data. For this purpose, a dataset of 10 48kHz audio samples, considered clean and unprocessed, was used. This dataset included private / proprietary datasets and public datasets such as VCTK (Y. Yamagishi, C. Veaux, and K. MacDonald, “CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice cloning toolkit (version 0.92)”, University of Edinburgh, Centre for Speech and Technology (CSTR), 2019 [online]: https: / / doi:org / 10:7488 / ds / 2645), RAVDESS (SRLivingstone and FA Russo, “The Ryerson audio-visual database of emotionalspeech and song (RAVDESS)” PLoS ONE [PLOS ONE], Vol. 13, No. 5, p. e0196391, 2018 [Online] Accessible: https: / / zenodo:org / record / 1188976 or TSP (http: / / www-mmsp:ece:mcgill:ca / Documents / Data / ). For the experiments of this disclosure, 50,000 quadruplets were used for training, 10,000 for validation, and 10,000 for testing. To form each quadruplet, the following was performed:

[0261] • Perform uniform sampling on the dataset, and then perform uniform sampling on the files within it.

[0262] • Uniformly sample frames at 1.1s intervals to avoid silent frames or primarily silent frames. Normalize them so that the maximum absolute amplitude is 1.

[0263] • With probabilities of 0.84, 0.12, and 0.04, zero, one, or two degradations are sampled from the available degradation pool (see below). If it is a zero-degradation, the signal is directly transformed into xi. Otherwise, we uniformly select the intensity for each degradation and apply them sequentially to generate x.i .

[0264] • Sample one, two, three, or four degradations from the available degradation pool with probabilities of 0.75, 0.2, 0.04, and 0.01 (see below). Uniformly select the intensity and apply it sequentially to x. i To generate x j .

[0265] • Uniformly sample time delays between 0ms and 100ms. Use such a delay from x i Extract 1s frame x ik and x il and for x i x in jk and x jl Perform the same operation.

[0266] • Store {x ik x il x jk x jl} as well as information on degradation type and intensity.

[0267] The total audio used is 78 hours: 1×4×(50000+10000+10000) / 3600=77:77h.

[0268] Appendix C: Degradation and Strength

[0269] Thirty-seven (37) possible degradations and their corresponding intensities were considered. The intensities were set such that they were perceptibly noticeable to the author at their minimum values. Note that in some cases, the intensities selected below represent only one aspect of the overall degradation, and for other relevant aspects, we sampled randomly from values ​​chosen empirically. For example, in the case of reverberation effects, SNR was chosen as the primary intensity, but reverberation type, width, delay, etc., were also randomly selected.

[0270] 1. Additional Noise – With a probability of 0.29, noisy frames are sampled from the available pool of noisy datasets. This is added to x, where the SNR is between 35dB and -15dB. Noise datasets include private / proprietary datasets and public datasets, such as ESC (KJPiczak, "ESC: dataset for environmental sound classification", Proc. of the ACM Conf. on Multimedia (ACM-MM), 2015, pp. 1015-1018 [available online: https: / / doi:org / 10:7910 / DVN / YDEPUT) or FSDNoisy18k (E. Fonseca, M. Plakal, DPWEEllis, F. Font, X. Favory, and X. Serra, "Learning sound event classifiers from web audio with noisy labels", ArXiv:1901.01189, 2019 [available online: https: / / doi:org / 10:5281 / zenodo:2529934). This degradation can be applied to the entire frame, or to a portion of it with a probability of 0.25 (at least 300ms).

[0271] 2. Colored Noise – Generate colored noise frames with a probability of 0.07 and a uniformity index between 0 and 0.7. Add this to x, where the SNR is between 45dB and -15dB. This degradation can be applied to the entire frame or to a portion of it with a probability of 0.25 (minimum 300ms).

[0272] 3. Buzzing sound – with a probability of 0.035, adds a tone of approximately 50Hz or 60Hz (sine, sawtooth, square) with an SNR between 35dB and -15dB. This degradation can be applied to the entire frame or to a portion thereof with a probability of 0.25 (minimum 300ms).

[0273] 4. Tone noise – probability 0.011, same as before but frequency between 20Hz and 12,000Hz.

[0274] 5. Resampling – with a probability of 0.011, the signal is resampled to a frequency between 2kHz and 32kHz and then converted back to 48kHz.

[0275] 6. μ-law quantization – with a probability of 0.011, μ-law quantization between 2 bits and 10 bits is applied.

[0276] 7. Clipping – with a probability of 0.011, clipping 0.5% to 99% of the signal.

[0277] 8. Audio inversion – probability 0.05, temporarily inverted signal.

[0278] 9. Insert silence – with a probability of 0.011, insert 1 to 10 silence segments with a length between 20ms and 120ms.

[0279] 10. Insertion noise – with a probability of 0.011, the same as above, but with white noise.

[0280] 11. Insertion attenuation – with a probability of 0.011, as above, but attenuating the segment by multiplying by the maximum linear gain of 0.8.

[0281] 12. Disturbance amplitude - probability is 0.011, same as above, but with inserted multiplicative Gaussian noise.

[0282] 13. Sample duplication – probability 0.011, same as above, but copies the previous sample.

[0283] 14. Delay – With a probability of 0.035, add a delayed version of the signal (single click and multiple clicks) with a maximum delay of 500ms.

[0284] 15. Extreme Equalization – Probability is 0.006, using an equalization filter with random Q and gain above 20dB or below -20dB.

[0285] 16. Bandpass – A bandpass filter with random Q applied at random frequencies between 100Hz and 4,000Hz with a probability of 0.006.

[0286] 17. Band stop – probability is 0.006, same as above, but it blocks the band.

[0287] 18. High-pass filter – with a probability of 0.011, applying a high-pass filter at random cutoff frequencies between 150Hz and 4,000Hz.

[0288] 19. Low-pass filter – with a probability of 0.011, applying a low-pass filter at a random cutoff frequency between 250 Hz and 8,000 Hz.

[0289] 20. Chorus - Probability is 0.011, add a chorus effect with a linear gain between 0.15 and 1.

[0290] 21. Overload - Probability is 0.011, adds an overload effect with a gain between 12dB and 50dB.

[0291] 22. Phase Shifter - With a probability of 0.011, adds a phase shifter effect with a linear gain between 0.1 and 1.

[0292] 23. Reverb - Probability 0.035, add reverb with an SNR between -5dB and 10dB.

[0293] 24. Tremolo - Probability is 0.011, adds a tremolo effect with a depth between 30% and 100%.

[0294] 25. Griffin-Lim Reconstruction – Perform Griffin-Lim reconstruction on the STFT of the signal with a probability of 0.023. The STFT is calculated using a random window length and 50% overlap.

[0295] 26. Phase randomization – with a probability of 0.011, the same as above, but with random phase information.

[0296] 27. Phase shuffling – probability is 0.011, same as above, but the phase of the window is shuffled in time.

[0297] 28. Spectral Convolution – STFT of the signal convolved with a 2D kernel at a probability of 0.011. The STFT is calculated using a random window length and 50% overlap.

[0298] 29. Spectral holes – probability 0.011, with a decrease in the application of spectral amplitude for probabilities between 0.15 and 0.98.

[0299] 30. Spectral noise – probability 0.011, same as above, but with a random value instead of 0.

[0300] 31. Transcode to MP3 – with a probability of 0.023, use libmp3lame and between 2kbps and 96kbps (all codecs are from ffmpeg) to encode to MP3 and return.

[0301] 32. Encode AC3 – with a probability of 0.035, encode to AC3 using either 2kbps or 96kbps and return.

[0302] 33. Encode EAC3 – with a probability of 0.023, encode to EAC3 using either 16kbps or 96kbps and return.

[0303] 34. Transcode to MP2 – with a probability of 0.023, encode to MP2 using a speed between 32kbps and 96kbps and return.

[0304] 35. WMA encoding – with a probability of 0.023, it encodes the data into WMA using a speed between 32kbps and 128kbps and returns the result.

[0305] 36. Encode OGG – with a probability of 0.023, use libvorbis and encode to OGG using 32kbps and 64kbps and return.

[0306] 37. Encode OPUS – with a probability of 0.046, use libopus and encode between 2kbps and 64kbps to OPUS and return.

[0307] Appendix D: Methods Considered

[0308] This disclosure is compared with nine existing methods:

[0309] 1. ITU-P563 (L. Malfait, J. Berger, and M. Kastner, “P.563 – The ITU-T standard for single-ended speech quality assessment”, IEEE Transactions on Audio, Speech and Language Processing, Vol. 14, No. 6, pp. 1924-1934, 2010) – This is a reference-free standard designed for narrowband telephones. This method was chosen because it is the best match for any reference-free standard available. The resulting scores are used directly.

[0310] 2. FL-JND – Inspired by Manocha et al. (P. Manocha, A. Finkelstein, Z. Jin, NJ Bryan, R. Zhang, and GJ Mysore, “Discriminative Perceptual Audio Metrics Learned from Minimal Perceptible Difference”, ArXiv:2001.04460, 2020), the proposed encoder architecture was implemented and trained on the JND task. Next, for each dataset, a small MLP was trained using sigmoid outputs that took latent features from all encoder layers as input and predicted quality scores.

[0311] 3. FL-PASE—The PASE encoder (S. Pascual, M. Ravanelli, J. Serrà, A. Bonafonte, and Y. Bengio, “Learning problem-agnostic speech representations from multiple self-supervised tasks”, Proc. of the Int. Speech Comm. Assoc. Conf. (INTERSPEECH), 2019, pp. 161-165) is trained using tasks such as JND, DT, and speaker recognition. Next, for each dataset, a small MLP is trained using a sigmoid output that takes latent features from the last layer as input and predicts a quality score.

[0312] 4. SRMR (THFalk, C. Zheng and W.-Y. Chan, “A non-intrusive quality and intelligibility measure of reverberant and dereverberated speech”, IEEE Transactions on Audio, Speech and Language Processing, Vol. 18, No. 7, pp. 1766-1774, 2010) – uses the metric from https: / / github:com / jfsantos / SRMRpy and employs a small MLP with sigmoid output to adapt it to the corresponding dataset.

[0313] 5. AutoMOS (B. Patton, Y. Agiomyrgiannakis, M. Terry, K. Wilson, R.A. Saurous, and D. Sculley, “AutoMOS: learning a non-intrusive assessor of naturalness-of-speech”, NIPS 16 End-to-end Learning for Speech and Audio Processing Workshop, 2016) – The method described was reimplemented, but the synthesized speech embedding and its auxiliary loss were replaced with L… MR .

[0314] 6. Quality-Net (S.-W. Fu, Y. Tsao, H.-T. Hwang and H.-M. Wang, “Quality-Net: an end-to-end non-intrusive speech quality assessment model based on BLSTM”, INTERSPEECH, 2018, pp. 1873-1877) – The proposed method has been reimplemented.

[0315] 7. WEnet (AA Cattellier and SD Voran, “WEnets: a convolutional framework for evaluating audio waveforms”, ArXiv:1909.09024, 2019) – The proposed method is suitable for regression MOS.

[0316] 8. CNN-ELM (H. Gamper, C.K.A.R. Eddy, R. Cutler, I.J. Tashev, and J. Gehrke, “Intrusive and non-intrusive perceptual speech quality assessment using a convolutional neural network”, IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), 2019, pp. 85-89) – The proposed method has been reimplemented.

[0317] 9. NISQA (G. Mittag and S. “Non-intrusive speech quality assessment for super-wideband speech communication networks”, Proc. of the IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 7125-7129 – The proposed method is suitable for working with MOS, and the auxiliary POLQA loss is replaced by L… MR .

Claims

1. A method for training a neural network-based system for determining an indication of the audio quality of an audio input, the method comprising: At least one training set including audio samples is obtained as input, wherein the audio samples include audio samples of a first type and audio samples of a second type, wherein each of the audio samples of the first type is labeled with information indicating a corresponding predetermined audio quality metric, and wherein each of the audio samples of the second type is labeled with information indicating a corresponding audio quality metric relative to a reference audio sample. The training set is input into the neural network-based system; and The system is iteratively trained to predict the corresponding label information of the audio samples in the training set. The training is based on multiple loss functions; and The multiple loss functions are generated to reflect the difference between the label information of the audio samples in the training set and their corresponding predictions.

2. The method according to claim 1, wherein, The first type of audio samples includes manually annotated audio samples, each manually annotated audio sample being labeled with information indicating the corresponding predetermined audio quality metric.

3. The method according to claim 2, wherein, The manually annotated audio samples include Mean Opinion Score (MOS) audio samples and / or Minimum Perceptible Difference (JND) audio samples.

4. The method according to claim 1, wherein, The second type of audio samples includes algorithm-generated audio samples, each labeled with information indicating a relative audio quality metric.

5. The method according to claim 4, wherein, Each of the samples generated by the algorithm is generated by selectively applying at least one degradation function, each having a corresponding degradation intensity, to a reference audio sample or an audio sample generated by another algorithm, wherein the tag information includes information indicating the corresponding degradation function and / or corresponding degradation intensity that has been applied to the reference audio sample or the audio sample generated by the other algorithm.

6. The method according to claim 5, wherein, The label information further includes information indicating degradation relative to the reference audio sample or other audio samples in the training set.

7. The method according to claim 5 or 6, wherein, The degradation function is selected from a plurality of available degradation functions, and / or one of them. The corresponding degradation intensity is set such that the degradation is perceptibly noticeable at its minimum value.

8. The method according to claim 7, wherein, The available degradation functions include functions related to one or more of the following: reverberation, clipping, encoding with different codecs, phase distortion, audio inversion, and background noise.

9. The method according to claim 5, wherein, The audio samples generated by the algorithm are used to create audio frame pairs. and / or audio frame quadruple Among them, audio frames It is generated by selectively applying at least one degradation function, each with a corresponding degradation intensity, to a reference audio frame, wherein the audio frame This is achieved by selectively applying at least one degradation function, each with a corresponding degradation intensity, to the audio frames. To generate, among which, audio frames and This is achieved by selectively applying a corresponding time delay to the audio frames. From the audio frame Extracted from, and among them, audio frames and This is achieved by selectively applying a corresponding time delay to the audio frames. From the audio frame Extracted from.

10. The method according to claim 1, wherein, The loss function includes a first loss function that indicates a MOS error metric, and wherein the first loss function is calculated based on the difference between the actual MOS data of the audio samples in the training set and the predictions for the audio samples.

11. The method according to claim 4, wherein, The label information of the second type of audio samples includes relative information indicating whether one audio sample degrades more than another audio sample, wherein the loss function includes a second loss function indicating a pairwise ranking metric, and wherein the second loss function is calculated based on the ranking established by the label information including the relative degradation information and its predictions.

12. The method according to claim 11, wherein, The system is trained in such a way that an audio sample with less degradation obtains an audio quality metric indicating better audio quality than another audio sample with more degradation.

13. The method according to claim 4, wherein, The label information of the second type of audio samples includes relative information indicating the perceptual relevance between audio samples, wherein the loss function includes a third loss function indicating a consistency index, and wherein the third loss function is calculated based on the difference between the label information including the perceptual relevance information and its prediction.

14. The method according to claim 13, wherein, The consistency index indicates whether two or more audio samples have the same degradation function and degradation intensity, and whether they correspond to the same time frame.

15. The method according to claim 4, wherein, The label information for the second type of audio samples includes relative information indicating whether an audio sample has been applied the same degradation function and the same degradation intensity as another audio sample, wherein the loss function includes a fourth loss function indicating a degradation condition index, and wherein the fourth loss function is calculated based on the difference between the label information, which includes relative degradation information, and its prediction.

16. The method according to claim 4, wherein, The label information of the second type of audio samples includes relative information indicating perceptual differences relative to each other, wherein the loss function includes a fifth loss function indicating the JND index, and wherein the fifth loss function is calculated based on the difference between the label information including relative perceptual differences and its prediction.

17. The method according to claim 4, wherein, The label information for the second type of audio samples includes information indicating the degradation function that has been applied to the audio samples, wherein the loss function includes a sixth loss function indicating a degradation type index, and wherein the sixth loss function is calculated based on the difference between the label information, which includes the corresponding degradation function information, and its prediction.

18. The method according to claim 4, wherein, The label information for the second type of audio sample includes information indicating the degradation intensity that has been applied to the audio sample, wherein the loss function includes a seventh loss function indicating a degradation intensity index, and wherein the seventh loss function is calculated based on the difference between the label information including the corresponding degradation intensity information and its prediction.

19. The method according to claim 1, wherein, The loss function includes an eighth loss function that indicates a regression index, wherein the regression index is calculated based on at least one of a reference-based quality metric and / or a referenceless quality metric.

20. The method according to claim 19, wherein, The reference-based quality metric includes at least one of the following: PESQ, CSIG, CBAK, COVL, SSNR, LLR, WSSD, STOI, SISDR, Mel cepstral distortion, and log-Mel band distortion.

21. The method according to claim 1, wherein, Each of the audio samples in the training set is used for at least one of the plurality of loss functions, and wherein the final loss function used for the training is generated based on an averaging process of one or more of the plurality of loss functions.

22. The method according to claim 1, wherein, The system includes an encoding level for mapping the audio input to a feature space representation and an evaluation level for generating predictions of label information based on the feature space representation.

23. The method according to claim 22, wherein, The encoding level used to generate the intermediate representation includes a neural network encoder.

24. The method according to claim 1, wherein, Each of the plurality of loss functions is determined based on a neural network including a linear layer or a multilayer perceptron (MLP).

25. A neural network-based system for determining an indication of the audio quality of an input audio sample, wherein, The system includes: Coding level; and Assessment level, The coding level is configured to map the input audio samples to a feature space representation; and The evaluation level is configured to predict information indicating a predetermined audio quality metric based on the feature space representation, and further predict information indicating a relative audio quality metric relative to a reference audio sample.

26. The system according to claim 25, wherein, The system is configured as follows: At least one training set is used as input, wherein the training set includes audio samples of a first type and audio samples of a second type, wherein each of the audio samples of the first type is labeled with information indicating a corresponding predetermined audio quality metric, and wherein each of the audio samples of the second type is labeled with information indicating a corresponding audio quality metric relative to the audio quality metric of the reference audio sample. Input the training set into the system; and The system is iteratively trained based on the training set to predict the corresponding label information of the audio samples in the training set based on multiple loss functions, wherein the multiple loss functions are generated to reflect the difference between the label information of the audio samples in the training set and their corresponding predictions.

27. A method for operating a neural network-based system to determine an indication of the audio quality of an input audio sample, wherein, The system includes an encoding level and an evaluation level, and the method includes: The input audio samples are mapped to a feature space representation by the coding level; and The evaluation level predicts information indicating a predetermined audio quality metric and information indicating a relative audio quality metric relative to a reference audio sample, based on the feature space representation.

28. A computer program product comprising instructions that, when executed by a processor, cause the processor to perform the steps of the method according to any one of claims 1 to 24 and 27.

29. A computer-readable storage medium storing a computer program product according to claim 28.

30. An apparatus comprising a processor and a memory coupled to the processor, wherein, The processor is adapted to cause the device to perform the steps of the method according to any one of claims 1 to 24 and 27.

Images