Machine listener-based audio streaming quality

Abstract

A method for evaluating playback performance in an adaptive streaming environment is provided. The method includes the steps of: acquiring playback-related information from a streaming client; estimating a representation of a test audio signal, which is an audio signal played back by the streaming client, based on the playback-related information; and determining an estimated value of the audio quality of the test audio signal based on the estimated representation of the test audio signal, using an audio quality evaluation algorithm. A method for providing playback-related information in a streaming client, as well as corresponding devices, programs, and computer-readable storage media, are also provided.

Description

[Technical Field]

[0001] Cross-references to related applications This application claims priority to U.S. Provisional Patent Application No. 63 / 497,941, filed on 24 April 2023, and European Patent Application No. 23181973.1, filed on 28 June 2023, each of which is incorporated herein by reference in whole.

[0002] Technical field This disclosure relates to a technique for evaluating playback performance in an adaptive streaming environment, and more particularly to a technique for constructing a deep neural network (DNN) for estimating subjective listening score indications for a test audio signal. [Background technology]

[0003] Algorithms for objective quality evaluation of coding schemes can be classified into two categories: so-called intrusive algorithms that require a reference signal in addition to the test signal, and non-intrusive algorithms that only analyze the test signal. Generally, intrusive algorithms perform better than non-intrusive algorithms. For example, in the context of lossy coding, it can be difficult to evaluate the performance of a codec without a reference, as the artistic intent is unknown in this situation (for example, a band-limited audio signal may not be an artifact from the codec, but rather the result of an intentional processing step applied in content creation). Furthermore, any metrics provided by non-intrusive quality evaluation tools can be difficult to interpret because such tools typically lack a basis in well-established subjective quality evaluation methodologies. For example, the Multi-Stimulus Test with Hidden Reference and Anchor (MUSHRA) is commonly used to evaluate audio codecs and involves the use of a reference signal.

[0004] Objective quality assessment can be particularly relevant in adaptive streaming environments where the quality of delivered content (e.g., bitrate) can be dynamically adjusted. The focus here is on the playback audio quality in each streaming client. However, since streaming clients typically lack access to the reference signal, intrusive algorithms are not applicable in this case.

[0005] Furthermore, listener scores achieved in listening tests such as the MUSHRA test can be predicted by a system that takes the test signal and reference signal as inputs.

[0006] For a given pair of input signals, listener scores exhibit some degree of variability, and capturing this aspect of the data for automated estimation of experience quality is valuable, for example, in entertainment delivery systems. In actual subjective MUSHRA trials with multiple listeners, the mean and standard deviation of listener scores can be calculated. The standard deviation can be converted into a confidence interval, given the number of listeners and the statistical model.

[0007] However, current automated administration of subjective listening tests like the MUSHRA test only provides the mean value of the MUSHRA score and cannot provide a concept of the variability of subjective listening scores that would be obtained from the population of test listeners.

[0008] Therefore, there is a need to improve the evaluation of audio quality in adaptive streaming environments. Further improvements are needed in the automated prediction of subjective listening scores. Techniques that can provide estimates of the variability of subjective listening scores in automated solutions are particularly needed. [Overview of the project] [Problems that the invention aims to solve]

[0009] In view of at least some of these needs, the Disclosure provides a method and apparatus for constructing a deep neural network (DNN) for estimating a subjective listening score index, a method for estimating a subjective listening score index using a DNN, a DNN, and corresponding apparatus, programs, and computer-readable storage media.

[0010] This disclosure further provides a method and apparatus for evaluating playback performance in an adaptive streaming environment, a method for providing playback-related information, and corresponding apparatus, programs, and computer-readable storage media. [Means for solving the problem]

[0011] Aspects of this disclosure relate to a method for configuring a DNN for estimating subjective listening score indications for an audio signal (e.g., a test audio signal). This method may, for example, be a method for training a DNN. The listening score may be a score obtained according to a listening test performed according to a predefined listening test method. The predefined listening test method may be a standardized listening test method. Furthermore, the listening test may apply predefined test metrics and / or test scenarios. The method may include providing an output stage of the DNN to generate listening score indications. The method may further include training the DNN in one of a plurality of training epochs by inputting one or more training data items that each represent a value for the listening score. Training the DNN may further include determining each listening score indication based on the one or more training data items in a training epoch. Training the DNN may further include determining each loss value for the one or more training data items by evaluating a loss function in a training epoch. Here, the loss function may depend on the listening score indication. Training the DNN may further involve adjusting one or more internal parameters of the DNN based on the determined loss value during the training epoch. The internal parameters of the DNN may be model parameters, such as coefficients for multiple layers of the DNN (e.g., filter coefficients).

[0012] Therefore, the DNN is trained on individual listening scores, rather than the mean of subjective listening scores. This makes it possible to adapt the DNN to predict parameters beyond the mean listening score, including, for example, the probability distribution, standard deviation, and / or confidence interval of the listening score.

[0013] In some embodiments, the instruction of the listening score may be related to the probability distribution of the listening score, and the output stage is adapted to generate the probability distribution of the listening score. The probability distribution may emulate the listening scores obtained by a plurality of listening tests for the audio signal. The plurality of listening tests emulated by the probability distribution may be independent listening tests. Further, the probability distribution may be parameterized by two or more parameters of the probability distribution. Then, determining each instruction of the listening score based on the one or more training data items may include determining each parameter of the probability distribution based on the one or more training data items. Determining the parameters of the probability distribution may be at least partially based on the value of the subjective listening score. Further, determining the parameters of the probability distribution may be based on the current state of the DNN, for example, the current values of the internal parameters of the DNN. The loss function may depend on the parameters of the distribution.

[0014] In some embodiments, training the DNN may be based on the maximum likelihood principle.

[0015] In some embodiments, the loss function may be related to the negative log likelihood (NLL) loss.

[0016] In some embodiments, the negative log likelihood loss may be given by L NLL =-log p θ (s|x,y). Here, p θ (s|x,y) is the probability distribution for the test score s when the representation of the audio signal y and the representation of the reference audio signal x for the audio signal y are given, and θ represents the internal parameters of the DNN.

[0017] By using the maximum likelihood principle, or correspondingly the negative log-likelihood loss, it becomes possible to efficiently train a DNN based on individual listening scores to provide indications of listening scores that could be expected in actual subjective listening tests with multiple listeners.

[0018] In some embodiments, the probability distribution has a mean μ and a variance σ. 2 It may also relate to a Gaussian distribution parameterized by the loss function L. Gauss teeth,

number

[0019] In some embodiments, the probability distribution may be a logistic distribution parameterized by mean μ and scale a, and the loss function L logistic teeth,

number

[0020] Both of these parameterizations of probability distributions have been shown to provide efficient training during the training stage and meaningful output during inference.

[0021] In some embodiments, the training data items may further represent the representation of the audio signal and the representation of the reference audio signal for the audio signal.

[0022] Therefore, the DNN under consideration may be configured to automate intrusive listening tests and possess the specific characteristics and advantages listed above.

[0023] In some embodiments, the representation of the audio signal and the representation of the reference audio signal may be related to a gammatone spectrogram.

[0024] Gammatone spectrograms are auditory features particularly well-suited to human hearing and perception, thus enabling the achievement of significant results with reduced computational complexity.

[0025] In some embodiments, the predefined listening test may be a Multi-Stimulus Test with Hidden Reference and Anchor (MUSHRA) listening test, such as the one standardized under ITU-R Recommendation BS.1534.

[0026] In some embodiments, the DNN may implement a generative model.

[0027] Another aspect of this disclosure relates to a method for estimating a subjective listening score indication for an audio signal using a DNN. The listening score may be a score obtained from a predefined listening test. The DNN may include an input stage for receiving a representation of the audio signal and a representation of a reference audio signal for the audio signal. The DNN may further include several layers for performing processing based on the representation of the audio signal and the representation of the reference audio signal. The processing by the several layers may further depend on the current state of the DNN, for example, the current values ​​of the DNN's internal parameters. The DNN may further include an output stage connected to the last of the several layers for generating a listening score indication. The method may include inputting the representation of the audio signal and the representation of the reference audio signal into the input stage. The method may further include determining the representation of the listening score indication based on the output of the output stage.

[0028] In some embodiments, the listening score index may be related to a probability distribution of listening scores, and the output stage of the DNN is adapted to generate a probability distribution of listening scores. The probability distribution may emulate listening scores obtained from multiple (subjective) listening tests on an audio signal. The probability distribution may be parameterized by two or more parameters of the probability distribution.

[0029] In some embodiments, determining the representation of the probability distribution may involve determining at least one of the mean, standard deviation, and confidence interval from the output of the output stage.

[0030] In some embodiments, the confidence interval may be determined based on the output of the output stage and the number of listeners (e.g., count) for the listening test being emulated.

[0031] In some embodiments, the probability distribution has a mean μ and a variance σ. 2 This may relate to a Gaussian distribution parameterized by μ or a logistic distribution parameterized by mean μ and scale a.

[0032] In some embodiments, the representation of the audio signal and the representation of the reference audio signal may be related to a gammatone spectrogram.

[0033] In some embodiments, the predefined listening test may be a MUSHRA listening test.

[0034] Another aspect of this disclosure relates to a DNN for estimating a subjective listening score indication for an audio signal. The listening score may be a score obtained from a predefined listening test. The DNN may include an input stage for receiving a representation of the audio signal and a representation of a reference audio signal for the audio signal. The DNN may further include several layers for performing processing based on the representation of the audio signal and the representation of the reference audio signal. The processing by the multiple layers may further depend on the current state of the DNN, for example, the current values ​​of the DNN's internal parameters. The DNN may further include an output stage coupled to the last of the multiple layers for generating a listening score indication.

[0035] In some embodiments, the DNN may be trained by inputting one or more training data items, each representing a different value of the listening score, during one of a plurality of training epochs. Training the DNN in a training epoch may further include determining each indication of the listening score based on the one or more training data items. Training the DNN in a training epoch may further include determining each loss value for the one or more training data items by evaluating a loss function, which may depend on the indication of the listening score.

[0036] Training a DNN in a training epoch may further involve tuning one or more internal parameters of the DNN based on the determined loss value. Constructing a DNN may involve, or correspond to, obtaining the internal parameters of the DNN by training it.

[0037] In some embodiments, the instruction for the listening score may be related to a probability distribution of the listening score, and the output stage of the DNN is adapted to generate a probability distribution of the listening score. The probability distribution may emulate listening scores obtained by multiple listening tests on an audio signal. Furthermore, the probability distribution may be parameterized by two or more parameters of the probability distribution.

[0038] In some embodiments, determining each indication of the listening score based on one or more training data items may include determining each parameter of a probability distribution based on one or more training data items. Determining the parameters of the probability distribution may be based at least in part on the subjective listening score values. Furthermore, determining the parameters of the probability distribution may also be based on the current state of the DNN, for example, the current values ​​of the DNN's internal parameters. The loss function may depend on the parameters of the distribution. In some embodiments, the probability distribution has a mean μ and a variance σ. 2 This can relate to a Gaussian distribution parameterized by μ, or a logistic distribution parameterized by mean μ and scale a.

[0039] In some embodiments, the representation of the audio signal and the representation of the reference audio signal may be related to (one or more) gammatone spectrograms. In some embodiments, the predefined listening test may be a MUSHRA listening test.

[0040] Another aspect of this disclosure relates to a device comprising a processor and memory coupled to the processor and storing instructions for the processor. The processor may be adapted to perform the aforementioned aspects and the methods according to their embodiments.

[0041] Another aspect of this disclosure relates to a device comprising a processor and memory coupled to the processor and storing instructions for the processor. The processor may be adapted to implement the aforementioned aspects and DNNs according to their embodiments.

[0042] Another aspect of this disclosure relates to a program that, when executed by a processor, includes instructions causing the processor to perform the aforementioned aspects and the methods according to their embodiments.

[0043] Another aspect of this disclosure relates to a program that, when executed by a processor, causes the processor to implement the aforementioned aspects and DNNs according to their embodiments.

[0044] Another aspect of this disclosure relates to a computer-readable storage medium storing one of the programs described above.

[0045] Another aspect of this disclosure relates to a method for evaluating playback performance in an adaptive streaming environment. Playback performance may relate to, for example, (subjective) playback quality. The method may include obtaining playback-related information from a streaming client. Playback-related information may include, correspond to, or be in the form of metadata. The method may further include estimating a representation of a test audio signal based on the playback-related information. The test audio signal may be an audio signal played back by a streaming client. Estimating a representation of the test audio signal may involve, or correspond to, reconstructing the test audio signal or a representation of it. The representation may relate to, for example, a set of features or spectrograms of the test audio signal. The method may further include using an audio quality evaluation algorithm to determine an estimate of the audio quality of the test audio signal based on the estimated representation of the test audio signal. The audio quality evaluation algorithm may be an objective audio quality evaluation algorithm. Furthermore, the audio quality evaluation algorithm may emulate an intrusive audio quality test (e.g., a listening test), such as the MUSHRA test. The estimated representation of the test audio signal is understood to be in a format suitable for input to an audio quality evaluation algorithm. For example, the representation of the test audio signal may relate to a predefined number of segments to accommodate the requirements of the audio quality evaluation algorithm related to the input audio signal or the time span covered by that representation.

[0046] This allows the proposed method to estimate the results of an intrusive listening test without requiring knowledge of the reference signal in the streaming client. When configured as described above, the intrusive listening test can be performed on a network node detached from the streaming client. The streaming client is only required to provide lightweight metadata to the network node performing the test. Both the version of the audio played and the reference for the audio played can be derived at the network node using the metadata. As a result, the proposed method can provide a significant and easily interpretable estimate of the audio quality of audio content played by a streaming client without significant additional signaling overhead to or from the streaming client.

[0047] In some embodiments, the method may further include generating a representation of a reference audio signal for a test audio signal. This may include obtaining (e.g., receiving) audio content or a representation thereof from a content repository. It is understood that the representation of the reference audio signal is in a format suitable for input to an audio quality evaluation algorithm. For example, the representation of the reference audio signal may relate to a predefined number of segments to accommodate the requirements of an audio quality evaluation algorithm related to the time span covered by the input audio signal or its representation.

[0048] In some embodiments, the method may further include obtaining an instruction from the streaming client for audio content processed by the streaming client. The instruction for audio content may include an identifier such as the file name of the audio content, the bitrate of the audio content, and / or information about the segmentation of the audio content. The instruction for audio content may be used to determine the sequence of audio content received by the streaming client. The audio content processed by the streaming client may be, for example, audio content received by the streaming client from a content delivery network.

[0049] In some embodiments, estimating the representation of the test audio signal may be based further on the instructions in the audio content.

[0050] In some embodiments, generating a representation of a reference audio signal may be based on an audio content instruction. This may include retrieving (e.g., receiving) the audio content or its representation from a content repository based on an audio content instruction to be played by a streaming client.

[0051] In some embodiments, playback-related information may include bitrate information indicating the bitrate of the audio signal to be played back by the streaming client. Estimation of the representation of the test audio signal may then be based on the bitrate information. Bitrate information may be provided for each of multiple segments of the audio signal to be played back by the streaming client.

[0052] In some embodiments, the audio quality evaluation algorithm may use a set of pre-trained models for audio quality evaluation. Generating an estimate of audio quality may involve selecting a pre-trained model from the set of pre-trained models based on playback-related information.

[0053] This ensures that the optimal model (for example, a model specifically trained for that situation) is used for each relevant situation, thereby improving the reliability of the regeneration performance estimate.

[0054] In some embodiments, playback-related information may include information related to the playback device associated with the streaming client. The pre-trained model may then be selected based on the playback device information. The playback device-related information may include indications of the playback device (e.g., headphones, soundbar, discrete speakers, etc.) and / or indications of the characteristics of the playback conditions (e.g., SNR, etc.).

[0055] Therefore, an appropriate model for audio quality evaluation may be used for each of several different playback device configurations, thereby improving the reliability of the playback performance estimation.

[0056] In some embodiments, the audio quality evaluation algorithm may be implemented by a deep neural network (DNN) for estimating subjective listening score indications for a representation of a test audio signal as an estimate of audio quality. The listening score may be a score from a predefined listening test. The DNN may include an input stage for receiving a representation of the test audio signal and a representation of a reference audio signal for the test audio signal. The DNN may further include several layers for performing processing based on the representation of the test audio signal and the representation of the reference audio signal. The DNN may further include an output stage for generating listening score indications.

[0057] In some embodiments, the DNN may be trained by inputting one or more training data items, each representing a value of a listening score, during one of a plurality of training epochs. The DNN may be further configured by determining each indication of the listening score based on the one or more training data items during the training epoch. The DNN may be further configured by determining each loss value for the one or more training data items by evaluating a loss function during the training epoch, the loss function depending on the indication of the listening score. The DNN may be further configured by adjusting one or more internal parameters of the DNN based on the determined loss values ​​during the training epoch.

[0058] In some embodiments, this method may be implemented on a network node different from the streaming client.

[0059] In some embodiments, the representation of the estimated value of the test audio signal may be associated with one or more gammatone spectrograms.

[0060] In some embodiments, the representation of the reference audio signal may be associated with one or more gammatone spectrograms.

[0061] In some embodiments, the method may further include outputting an estimate of the audio quality of the test audio signal to a network node different from the network node associated with the streaming client. The network node may be, for example, a network node within a cloud-based framework.

[0062] In some embodiments, an estimate of the audio quality of a test audio signal may be output to a network node to perform encoding and / or packaging of audio content. The method may then further include optimizing the encoding and / or packaging based on the estimate of the audio quality of the test audio signal.

[0063] In some embodiments, the method may further include determining an optimal number of quality levels in a bitrate ladder for delivery by a content delivery network, based on an estimate of the audio quality of a test audio signal. This may involve, for example, inputting an estimate of the audio quality of a test audio signal into a utility function used to determine the optimal number of quality levels.

[0064] In some embodiments, the method may further include determining the configuration and / or set of encoding tools based on an estimate of the audio quality of the test audio signal.

[0065] In some embodiments, the method may further include determining an estimate of the audio quality of a test audio signal for each of a population of streaming clients. The method may further include comparing the determined audio quality estimates for the population of streaming clients.

[0066] This could allow for the comparison of different content delivery and / or playback methods to determine the optimal delivery and / or playback method.

[0067] Another aspect of this disclosure relates to a method for providing playback-related information in a streaming client processing audio content in an adaptive streaming environment. The method may include generating playback-related information by analyzing a playback buffer associated with the streaming client to determine bitrate information indicating the bitrate of a segment of an audio signal to be played back by the streaming client; analyzing manifest information associated with the audio content; and analyzing the characteristics of a playback device associated with the streaming client. The method may further include outputting the playback-related information to a network node different from the network node associated with the streaming client.

[0068] Another aspect of this disclosure relates to a device comprising a processor and memory coupled to the processor and storing instructions for the processor. The processor may be adapted to perform methods according to either of the two aforementioned aspects and any one of their embodiments.

[0069] Another aspect of the present invention relates to a program, which, when executed by a processor, includes instructions that cause the processor to perform the two aforementioned aspects and the methods according to their embodiments.

[0070] Another aspect of this disclosure relates to a computer-readable storage medium storing the program of the aforementioned aspect.

[0071] It should be noted that methods and systems, including preferred embodiments thereof as outlined herein, may be used independently or in combination with other methods and systems disclosed herein. Furthermore, all aspects of the methods, apparatus, and systems outlined herein may be combined in any way.

[0072] In particular, the features of the claims may be combined with each other in any way.

[0073] It will be understood that the features of the apparatus and the steps of the method can be replaced in many ways. In particular, the details of the disclosed method can be implemented by the corresponding apparatus, as those skilled in the art will understand, and vice versa. Furthermore, it will be understood that any of the above statements made with respect to the method (and, for example, its steps) are equally applicable to the corresponding apparatus (and, for example, its blocks, stages, units), and vice versa. [Brief explanation of the drawing]

[0074] The present invention is described below illustratively with reference to the accompanying drawings.

[0075] In the drawings, the same numbers represent the same elements.

[0076] [Figure 1] This block diagram schematically illustrates an example of a framework for evaluating playback performance in an adaptive streaming environment according to an embodiment of the present disclosure.

[0077] [Figure 2] This flowchart shows an example of a method for evaluating playback performance in an adaptive streaming environment according to embodiments of the present disclosure.

[0078] [Figure 3] A is a block diagram schematically illustrating playback analysis and generation of playback-related information in a streaming client according to an embodiment of the present disclosure. B is a block diagram schematically illustrating model selection for evaluating playback performance according to an embodiment of the present disclosure.

[0079] [Figure 4] A and B are flowcharts illustrating examples of methods for generating playback-related information according to embodiments of the present disclosure.

[0080] [Figure 5]This block diagram schematically illustrates an example of evaluating regeneration performance using pre-calculated data according to an embodiment of the present disclosure.

[0081] [Figure 6] An example of a graphical user interface for representing raw performance according to an embodiment of this disclosure is schematically shown.

[0082] [Figure 7] This flowchart shows an example of using evaluated playback performance to compare a population of streaming clients according to embodiments of the present disclosure.

[0083] [Figure 8] This block diagram schematically illustrates an example of a framework for comparing different content delivery networks according to embodiments of the present disclosure.

[0084] [Figure 9] This block diagram schematically illustrates an example of a framework for comparing different content delivery or playback methods according to embodiments of the present disclosure.

[0085] [Figure 10] This block diagram schematically illustrates an example of a framework for optimizing content encoding and / or packaging in an adaptive streaming environment according to embodiments of the present disclosure.

[0086] [Figure 11] This flowchart shows an example of a method for optimizing content encoding and / or packaging in an adaptive streaming environment according to embodiments of the present disclosure.

[0087] [Figure 12] This is a schematic block diagram illustrating an example of a DNN for estimating subjective listening score indications according to an embodiment of the present disclosure.

[0088] [Figure 13] This flowchart illustrates an example of a method for configuring a DNN for estimating subjective listening scores according to embodiments of the present disclosure. [Figure 14] This flowchart illustrates an example of a method for configuring a DNN for estimating subjective listening scores according to embodiments of the present disclosure.

[0089] [Figure 15] This flowchart shows an example of a method for estimating subjective listening score indications using a DNN according to embodiments of the present disclosure.

[0090] [Figure 16] This figure shows an example of the performance of a trained or partially trained DNN according to an embodiment of the disclosure. [Figure 17] This figure shows an example of the performance of a trained or partially trained DNN according to an embodiment of the disclosure. [Figure 18] This figure shows an example of the performance of a trained or partially trained DNN according to an embodiment of the disclosure. [Figure 19] This figure shows an example of the performance of a trained or partially trained DNN according to an embodiment of the disclosure. [Figure 20] This figure shows an example of the performance of a trained or partially trained DNN according to an embodiment of the disclosure.

[0091] [Figure 21] This block diagram shows an example of an apparatus for performing a method or implementing a DNN according to embodiments of the present disclosure. [Modes for carrying out the invention]

[0092] This disclosure describes, in order, techniques for estimating the audio quality of content played in an adaptive streaming environment, and techniques for configuring (e.g., training) and using a DNN for estimating audio quality.

[0093] Machine listener-based audio streaming quality Broadly speaking, part of this disclosure relates to techniques (e.g., methods, apparatus, and systems) for performing objective quality testing of audio, for example, in the context of content streaming over the Internet (e.g., adaptive streaming, particularly adaptive audio streaming). A system implementing such techniques may include a cloud-based service receiving playback-related information (e.g., playback-related parameters) from a client (streaming client) or a set of clients (i.e., a population of clients), and then calculating an objective quality score by emulating a subjective quality assessment test, for example, according to the MUSHRA method (standardized under ITU-R Recommendation BS.1534).

[0094] The proposed techniques facilitate the evaluation of audio quality from encoding to actual client playback. Furthermore, these methods allow for the evaluation of the client population in relation to the audio quality delivered to clients.

[0095] Furthermore, the proposed technique can be used for monitoring the audio experience. This technique can also be used to perform A / B testing (e.g., bucket or split-run testing) on ​​a real-world population of streaming clients (e.g., for experiments comparing bitrate ladder and / or codec performance). The system can perform quality analysis in online and offline settings. In online mode, audio quality can be estimated in real time as the streaming progresses, based on a feedback channel between the client and the service. In offline settings, the service can first collect all parameter data from clients participating in the experiment, and then perform analysis based on that data.

[0096] In particular, the proposed technique may be applied to the so-called Generative Machine Listener, which will be described later in this disclosure. As will be described in more detail later, the Generative Machine Listener is a neural network (e.g., DNN) trained to evaluate audio by comparing it to a reference signal and to provide the evaluation results as a probability distribution having, for example, a mean corresponding to a MUSHRA score and a confidence value corresponding to the confidence interval expected in a listening test performed on such material.

[0097] definition Intrusive quality assessment This requires access to both a reference signal and a test signal. Established subjective test methods use this approach. There are non-intrusive quality methods that require only access to the test signal. However, these are unreliable, and the results can be difficult to interpret.

[0098] Objective quality assessment algorithmThis facilitates the estimation of perceived quality by human observers without the use of human observers. For example, a machine listener performs objective quality assessment by predicting the quality score that would be achieved in a subjective test by a human observer. In particular, the machine listener facilitates the estimation of mean performance scores, for example, along with the relevant confidence intervals.

[0099] Adaptive streaming Adaptive streaming is a content delivery method in which content is available in multiple quality versions associated with different bitrates. Higher bitrates generally correspond to higher quality. The content player includes a policy that attempts to determine the best possible bitrate, which will result in the delivery of segments in time before they are due to be played. In other words, adaptive streaming policies attempt to maximize the quality of the experience (for example, in a setting where content segments are downloaded, inserted into the playback buffer, and played) while keeping the probability of playback buffer exhaustion below some reasonably low threshold (i.e., the probability of re-buffering remains low).

[0100] Bitrate ladder A bitrate ladder is a set of content versions associated with different quality versions of the content. The bitrates of the content within a bitrate ladder are designed to facilitate streaming across diverse throughput scenarios (e.g., from very low throughput to very high throughput). Adaptive streaming policies select the appropriate quality level from the bitrate ladder for each segment. Information about the bitrate ladder is typically supplied to the content player in a so-called manifest.

[0101] Description of Exemplary Embodiments Figure 1 shows an example of a quality assessment service that incorporates a machine listener (for example, as part of a quality assessment service) within a framework for adaptive streaming. The machine listener facilitates intrusive quality assessment of the audio experience at the client level without requiring the client to provide unencoded references.

[0102] As mentioned above, using intrusive algorithms for quality assessment in (adaptive) streaming environments is hindered by the fact that streaming clients typically do not have access to a reference signal. Providing a reference signal to a streaming client typically requires out-of-band placement of the reference signal, which is highly undesirable due to bandwidth limitations.

[0103] Embodiments of this disclosure facilitate the execution of intrusive testing, for example, within a cloud or network service, at nodes (test nodes, network nodes) from which reference signals may be supplied. Instead of sending client replay signals to an upstream test node, the test signals are reassembled at the test node based on lightweight replay-related metadata. Lightweight replay-related metadata may be acquired by an instrumented client and then transmitted to the upstream service.

[0104] In the example in Figure 1, the streaming client 10 receives encoded audio content 5 (e.g., audio content, or video content with associated audio content) from a Content Delivery Network (CDN) 105, for example, over the internet. The CDN 105 may provide different versions of a given content, for example, at different bitrates (e.g., using different settings within a predefined bitrate ladder), depending on the streaming client configuration and / or network conditions. On the other hand, the streaming client 10 may be configured to employ adaptive bitrate control to request content at different bitrates in order to maximize playback quality and / or user experience.

[0105] After appropriate decoding, the streaming client 10 plays the audio content through a playback buffer, for example, segment by segment. Simultaneously, the streaming client 10 performs playback analysis, for example, through a playback analysis block 120, to generate playback-related information 20 (e.g., playback-related metadata, or playback metadata). In this sense, the streaming client 10 functions as, implements, or has an instrumented client that collects and transmits the playback-related information 20. The playback-related information 20 is provided to or retrieved by a quality evaluation service 150 (e.g., a machine listener service). Generally, the playback-related information 20 is sometimes said to be provided to or retrieved by a test node. Furthermore, instructions for the audio content processed by the streaming client 10 are provided to or retrieved by the quality evaluation service 150 (or a test node), enabling the quality evaluation service 150 to generate reference signals related to the content processed by the streaming client 10 for intrusive quality evaluation. Here, the audio content instruction may include identifiers such as a file name (e.g., an audio segment file name), the audio content bitrate level, and / or information about the (current) segmentation of the audio content. The audio content instruction may be used by the streaming client 10 to determine the sequence of audio segments to be received and played. The audio content processed by the streaming client 10 may be audio content received by the streaming client 10 from, for example, a CDN 105. The audio content instruction may be obtained, for example, by intercepting the streaming client 10's request to the CDN 105.

[0106] The quality evaluation service 150 may be in the form of a network service or a cloud service. Furthermore, the quality evaluation service 150 may be configured to perform a method for evaluating playback performance in an adaptive streaming environment, such as method 200 described later. For this purpose, the quality evaluation service 150 may have a trained network 40 and a model selector 145 for selecting a model that is appropriately trained from a set of models based on playback-related information 20. The quality evaluation service 150 may further have a test signal reproduction block 130 for estimating a test signal 30 and a reference lookup block 160 for estimating a reference signal 60 for the test signal 30.

[0107] An example of a method 200 for evaluating playback performance in an adaptive streaming environment (for example, by the quality evaluation service 150 in Figure 1) is shown in the flowchart of Figure 2. Playback performance may relate to, for example, (subjective) playback quality. Method 200 includes steps S210 to S230, as well as an optional step S240. This method may be implemented on a network node different from the streaming client 10 (for example, a test node). Furthermore, it may be implemented on a network node different from the CDN 105. Here, the network node may be, for example, a network node in a cloud-based framework.

[0108] Step S210 Playback-related information is then obtained from the streaming client. This playback-related information may include, correspond to, or be in the form of metadata.

[0109] Step S220In this process, the representation of the test audio signal is estimated based on playback-related information. Here, the test audio signal is the audio signal played back by the streaming client. Estimating the representation of the test audio signal may involve or correspond to reconstructing the test audio signal or its representation. The representation may relate, for example, to a set of features or spectrograms (e.g., a gammatone spectrogram) of the test audio signal.

[0110] Step S230 In this system, an estimate of the audio quality of the test audio signal is determined using an audio quality evaluation algorithm based on an estimated representation of the test audio signal. The audio quality evaluation algorithm can be an objective audio quality evaluation algorithm. Furthermore, the audio quality evaluation algorithm can emulate intrusive audio quality tests (e.g., listening tests), such as the MUSHRA test. It is understood that the estimated representation of the test audio signal is in a format suitable for input to the audio quality evaluation algorithm. For example, the representation of the test audio signal may relate to a predefined number of segments to address the requirements of the audio quality evaluation algorithm related to the time span covered by the input audio signal or its representation.

[0111] It may be optional Step S240 In this scenario, the estimated audio quality of the test audio signal is output to a network node different from the network node associated with the streaming client. Non-exclusive examples of using the estimated audio quality of the test audio signal are described below with reference to Figures 6, 7, 8, 9, 10, and 11.

[0112] Configured as described above, the technique provided by this disclosure uses a quality assessment service (e.g., including a machine listener) as a cloud service for streaming quality assessment. The quality assessment service (e.g., a machine listener) is a component independent of the content delivery system and independent of the streaming client. The quality assessment service (e.g., a machine listener) has the following attributes: This allows the client device to perform intrusive quality testing of the playback without needing to supply a reference signal to the client device. This allows for the implementation of objective quality assessments that can emulate well-established subjective testing methods, such as MUSHRA. By appropriately selecting an audio quality evaluation algorithm, results can be presented as a probability distribution (e.g., MUSHRA score + confidence interval).

[0113] Potential uses and benefits of the techniques according to embodiments of this disclosure may include: The techniques according to embodiments of this disclosure facilitate delivery using a typical CDN and allow the quality evaluation system to be decoupled from the CDN infrastructure, which can be advantageous. Thus, the CDN generally does not need to be involved in operating the machine listener service and does not need to store reference signals in the CDN. The proposed techniques also facilitate multi-CDN delivery. The techniques according to embodiments of this disclosure facilitate experiments such as the evaluation of scenarios in which quality estimates cannot be precalculated. One example of such a scenario is when the number of ABR ladder level combinations in the playback buffer and the number of different ways of playback (e.g., speakers in handheld devices, headphones, discrete speakers) are disproportionately large. An exemplary application of the system shown in the example in Figure 1 is A / B testing of bitrate ladders or codecs in real-world content delivery scenarios (e.g., in a real population of clients across actual content delivery conditions).

[0114] Systems and methods according to embodiments of the present disclosure, configured as described above, include means / steps that enable the emulation of an intrusive listening test (e.g., a MUSHRA test) by operating a (generating) machine listener in the cloud, the test signal being reconstructed (or partially reconstructed) within the service by using a feedback channel from an instrumentation client transmitting playback-related metadata. Furthermore, these systems and methods involve the selection of an appropriate model to be used by the (generating) machine listener from a set of pre-trained models based on the playback-related metadata.

[0115] In addition to the above, Method 200 may also include a step (not shown in Figure 2) of generating a representation of a reference audio signal for use by an audio quality evaluation algorithm. This may include obtaining (e.g., receiving) audio content or a representation of it from a content repository (or generally a content source). It is understood that the representation of the reference audio signal should be in a format suitable for input to the audio quality evaluation algorithm. For example, the representation of the reference audio signal may relate to a predefined number of segments to address the requirements of the audio quality evaluation algorithm related to the time span covered by the input audio signal or its representation.

[0116] In addition to the playback-related information 20, the quality evaluation service 150 may further require information (e.g., identification information) about the audio content processed by the streaming client 10. Therefore, method 200 may further include a step (not shown in Figure 2) of obtaining information about the audio content processed by the streaming client 10 from the streaming client 10. As described above, the information about the audio content may include an identifier such as the file name of the audio content, the bitrate level of the audio content, and / or information about the segmentation of the audio content. The information about the audio content may be used to determine the sequence of audio segments received by the streaming client 10. The audio content processed by the streaming client 10 may be audio content received by the streaming client 10 from, for example, a CDN 105. Then, using the available information about the audio content, estimating the representation of the test audio signal in step S220 may further rely on the information about the audio content. Furthermore, the generation of a representation of a reference audio signal may also be based on the aforementioned information about the audio content. This may include retrieving (for example, receiving) audio content or a representation thereof from a content repository based on the aforementioned instructions for audio content to be played by the streaming client.

[0117] If the playback-related information 20 includes bitrate information indicating the bitrate of the audio signal to be played back by the streaming client 10, then estimating the representation of the test audio signal in step S220 may be based on the bitrate information. This bitrate information may be provided for each of the multiple segments of the audio signal to be played back by the streaming client 10.

[0118] An exemplary process for estimating the representations of the test signal and the reference signal is described below with reference to Figure 5.

[0119] Figure 3A shows an example of a streaming client 10 that collects playback-related information (e.g., playback-related metadata), which is then aggregated and transmitted upstream to facilitate the operation of a quality evaluation service 150 (e.g., a machine listener service). Thus, Figure 3A relates to the operation of an instrumentation client implemented by or included in the streaming client 10.

[0120] An example of a corresponding method 400 for providing playback-related information in a streaming client processing audio content in an adaptive streaming environment is shown in the flowchart of Figure 4A. Method 400, performed in the streaming client, includes steps S410 and S420. Method 450, shown in the flowchart of Figure 4B, relates to step S410 in more detail. Method 450 includes steps S460 to S480.

[0121] Step S410 Then, playback-related information is generated.

[0122] Step S420 In this case, playback-related information is output to a network node different from the network node associated with the streaming client (for example, a test node). For example, as mentioned above, playback-related information may be output to the quality evaluation service 150.

[0123] Method 400 may further include the step (not shown) of providing instructions for audio content processed by a streaming client, as described above in the context of Figure 1.

[0124] Steps S460-S480 of Method 450 in Figure 4B relate to the details and possible implementations of step S410 of Method 400. It is understood that step S410 may include one or more, potentially all, of steps S460-S480.

[0125] Step S460 The playback buffer associated with the streaming client is then analyzed to determine bitrate information, which indicates the bitrate of the segments of the audio signal played back by the streaming client. Thus, the bitrate information can indicate the bitrate of each of the multiple sequential segments of the played content. In addition to bitrate information, the analysis of the playback buffer can also yield further information about the configuration of the playback buffer.

[0126] Here, it is understood that the playback buffer typically contains a sequence of segments. Bitrate changes may occur on a segment-by-segment basis, for example, due to actions of an ABR policy running on the streaming client.

[0127] Step S470 Next, manifest information related to the audio content is analyzed. Analyzing the manifest information can provide information about the bitrate ladder currently in use.

[0128] Step S480 Next, the characteristics of the playback device associated with the streaming client are analyzed. The characteristics of the playback device may relate to the device type and / or the type of playback system used, for example, whether it is headphone playback or speaker playback.

[0129] Therefore, returning to A in Figure 3, for example, the playback analysis by playback analysis block 120 may include one or more of the following: playback buffer analysis (for example, in playback buffer analysis block 310), manifest analysis (for example, in manifest analysis block 320), and playback device analysis (for example, in playback device analysis block 330).

[0130] Furthermore, in accordance with the above, playback-related information (e.g., metadata) may include information regarding the player buffer configuration (determined, for example, by playback buffer analysis), the currently used bitrate ladder configuration (determined, for example, by manifest analysis), and / or information regarding the identification of the playback device (determined, for example, by playback device analysis).

[0131] The operation and characteristics of the instrumentation client related to the streaming client 10 can be briefly summarized as follows: The techniques described herein require an instrumentation client (which can be easily uploaded to a client device), but do not require any other actions (such as placing a reference on the client). • Playback analysis extracts playback-related information (e.g., the sequence of segments in the playback buffer, information about the playback device, information about the manifest content) and sends it upstream as metadata. Metadata can be used to reproduce the test signal or its features (e.g., a spectrogram such as a gammatone spectrogram) and to perform a lookup of a relevant reference signal. Given two signals (i.e., a test signal and a reference signal), the quality assessment service 150 can calculate an indication of a quality score (e.g., a probability distribution representing the quality score) when selecting a properly trained model.

[0132] Furthermore, potential applications and benefits of the instrumentation client and / or its operation may include: The system can distinguish between different playback scenarios. For example, headphone playback may, in some cases, be more critical than speaker playback. This may be reflected in the quality score generated by the client. To achieve this, a cloud service that performs quality assessments may include a set of pre-trained models. For example, there may be a model trained on listening test data from tests performed on various headphones. There may also be another model trained on listening tests performed on discrete speakers. Since the two playback scenarios generally differ in how critical they are, it may be beneficial to use dedicated models for these scenarios and then use the appropriate model to perform the playback quality assessment.

[0133] Figure 3B is a schematic block diagram illustrating an exemplary process for selecting a model (from a set of pre-trained models) that can be used by a quality evaluation service 150 (e.g., a machine listener). Model selection is based on playback-related information 20 transmitted upstream by the instrumentation client 10.

[0134] For example, different models may be trained on different training data related to their respective different use cases. In one embodiment, different models may be trained on different device characteristics, such as different device types and / or different playback systems (e.g., headphones or speakers).

[0135] Model selection may be performed in model selection block 145. The selection may be made from a set of models 340, which includes individual models 340-1, 350-2, 350-3, ... Each of these models may be associated with a machine listener or DNN trained for audio quality evaluation under specific circumstances. For example, the models in the set of models 340 may be trained for different device characteristics, as described above.

[0136] Returning to method 200 in Figure 2, the audio quality evaluation algorithm employed by this method may use a set of pre-trained models for audio quality evaluation. Then, in step S203, determining (e.g., generating) an estimate of audio quality may include selecting a pre-trained model from the set of pre-trained models based on playback-related information.

[0137] For example, as mentioned above, playback-related information 20 may include information related to the playback device associated with the streaming client. The pre-trained model may then be selected based on the playback device information. The playback device-related information may include instructions for the playback device (e.g., headphones, soundbar, discrete speakers, etc.) and / or instructions for the characteristics of the playback conditions (e.g., SNR, etc.).

[0138] In some embodiments, the quality evaluation service may relate to a generating machine listener (e.g., a stereo generating machine listener) implemented by a DNN, using algorithms described later under the section "Machine Listener" (e.g., the audio quality evaluation algorithm described above). Instead of operating directly on waveforms, this algorithm may operate on test signals and reference signals, or on spectrograms calculated for those signals (e.g., gammatone spectrograms). This means that both the test signals and reference signals can be assembled from pre-calculated blocks (e.g., pre-calculated blocks of spectrograms), which can reduce cloud storage requirements and computational load. This is particularly advantageous in terms of reducing the cost of operating a machine listener in the cloud.

[0139] Spectrograms (or other audio features) can be calculated segment by segment according to the segmentation introduced by the transport mechanism used by the content delivery system (e.g., by CDN 105).

[0140] Therefore, a (stereo) generating listener can operate on gammatone spectrograms of the left, right, center, and side signals of the reference signal and the encoded stereo signal (as described, for example, in Patent Document 1). Gammatone filtering is a general approximation of filtering performed by the ear. Thus, gammatone-based spectrograms can be considered more perceptually motivated representations than traditional spectrograms. Gammatone spectrograms of audio signals can be calculated using, for example, an 80ms window size, a 20ms hop size, and 32 frequency bands ranging from 50Hz to 24kHz. The resulting gammatone spectrograms of short segments of the reference signal and the encoded signal (e.g., a 1-second signal in an ABR ladder) can be pre-calculated, paired, and stacked along the channel dimension, which results in, for example, an input size of 8 × 32 × 50 (channels × bands × time frames) to a neural network.

[0141] Therefore, in some embodiments, the aforementioned representations of the estimated value of the test audio signal and the aforementioned representations of the reference audio signal may each relate to one or more gammatone spectrograms (e.g., left (L), right (R), center (M), and side (S) spectrograms).

[0142] However, it should be understood that the techniques described herein (e.g., methods and apparatus) are not limited to the use of spectrograms (e.g., gammatone spectrograms), and these techniques may work similarly with waveforms or other audio features. Furthermore, spectrograms other than gammatone spectrograms, such as other perceptually motivated spectrograms, may be used for this purpose. However, for the sake of brevity of presentation, and without limitation, the following will refer to spectrograms, and in particular gammatone spectrograms, instead of general spectrograms or waveforms.

[0143] Figure 5 schematically illustrates the process of assembling a reference signal and reconstructing a test signal based on playback-related information 20 (e.g., metadata) transmitted by the instrumentation client 10 to the upstream quality evaluation service 150. In some embodiments, the data flow process used to feed the quality evaluation model (e.g., a generating listener model such as the trained network 40 in Figure 1) may include several pre-calculated steps.

[0144] Pre-calculated spectrograms (e.g., gammatone spectrograms) are maintained in the reference repository 580 for each segment. Based on the instructions for the audio content (segments) played by the streaming client 10 (e.g., the ID and segmentation of the played content item), the reference repository 580 is queried by the reference lookup block 585 in the reference assembly block 590 for the assembly of a reference signal for each segment. The assembled reference signal 560 is provided to an audio quality evaluation algorithm, such as a machine listener, for audio quality evaluation in the machine listener analysis block 540.

[0145] Furthermore, playback-related information 20 is provided to the test signal assembly block 575 along with instructions for the audio content (segments) to be played back by the streaming client 10 (e.g., ID and segmentation of the content items to be played back). Based on the playback-related information 20 (e.g., bitrate, codec configuration), the content repository 570 is again queried to assemble the test signal segment by segment. This provides a representation of the aforementioned test audio signal 530 for audio quality evaluation algorithms, e.g., for input to the machine listener, for audio quality evaluation in the machine listener analysis block 540. The assembly of the representation of the test audio signal 530 may correspond, for example, to step S220 of method 200.

[0146] Although not shown in Figure 5, it is understood that playback-related information 20 may also be provided to the machine listener analysis block 540 for the model selection described above.

[0147] Based on the assembled representation of the test audio signal 530 and the assembled reference signal 560, the audio quality evaluation algorithm can generate an estimate of the audio quality of the test audio signal, as described above with reference to step S230 of Method 200.

[0148] Figure 6 is a non-limiting example of a graphical user interface illustrating the results provided by the technique according to embodiments of the present disclosure. The GUI includes an indicator / selector of available levels for a bitrate ladder 610–650, as well as an audio score indicator 660 for the test signal. This indicator 660 may include, for example, the mean and confidence interval for a subjective listening score, such as a MUSHRA score.

[0149] Downstream applications for audio quality estimation The techniques described herein can operate in both online and offline settings.

[0150] In online setup, quality analysis may be performed on the fly, and performance scores (e.g., an estimate of audio quality) can be delivered to the delivery system whenever required.

[0151] The offline setting involves aggregating playback-related information (e.g., playback metadata) from multiple streaming clients (e.g., two sets of clients used for A / B testing). The test signal can be constructed offline (e.g., after the experiment is completed) from the collected playback-related information. A quality assessment service (e.g., a machine listener) can then perform quality analysis offline and provide performance statistics for the client (or set of clients).

[0152] In general, regardless of whether an online or offline setting is applied, the techniques described herein can be used to compare streaming clients within different groups of streaming clients, or to compare different groups of streaming clients.

[0153] An example of the corresponding method 700 is schematically shown in the flowchart of Figure 7. Method 700 includes steps S710 and S720 and may be performed following or in connection with method 200 described above.

[0154] Step S710 Then, estimates of the audio quality of the test audio signal of the streaming client are determined for each of the multiple populations of streaming clients. This can be done as described above in the context of Method 200.

[0155] Step S720 Next, the estimated audio quality values ​​determined for the aforementioned population of streaming clients are compared with each other. Generally, the determined estimates are analyzed.

[0156] Examples of such comparisons, applications, and use cases are schematically shown in the block diagrams of Figures 8 and 9.

[0157] In the example in Figure 8, the content server 850 (content origin) provides the respective (audio) contents 852 and 854 to the first and second content delivery networks CDN1, 830, and CDN2, 840. The first CDN 830 provides content 835 to the first group 810 of streaming clients 815. The second CDN 840 provides content 845 to the second group 820 of streaming clients 825. The streaming clients of both populations 810 and 820 provide their respective sets of playback-related information 870 and 880 to the quality evaluation service 860, which, using the techniques described above, determines the respective estimates of audio quality 865 (or generally, estimates of playback performance) for the populations 810 and 820 based on the respective sets of playback-related information 870 and 880. Determining each estimated value of audio quality 865 may require receiving the downloaded segment 855 or its representation (provided to the streaming client) from the content server 850. Comparing estimated values ​​of audio quality 865 for two populations 810 and 820 allows for inferences about the different performance of different CDNs 830 and 840, for example. This information can be used to optimize content delivery to a population of streaming clients.

[0158] In the example in Figure 9, the content server 950 (content origin) provides (audio) content 952 to the CDN 910, the CDN 920 provides content 932 to the first group 910 of streaming clients 915, and content 934 to the second group 920 of streaming clients 925. The streaming clients of both populations 910 and 920 provide their respective sets of playback-related information 970 and 980 to the quality evaluation service 960, which, using the techniques described above, determines their respective estimates of audio quality 965 (or generally, estimates of playback performance) for populations 910 and 920 based on their respective sets of playback-related information 970 and 980. Determining their respective estimates of audio quality 965 may require receiving the downloaded segment 955 or its representation (provided to the streaming clients) from the content server 950. By comparing estimated audio quality values ​​for two populations of streaming clients, 910 and 920, it becomes possible to infer information about the different performance of different populations, for example, when different delivery and / or playback methods are employed for / by different populations. This information can be used to optimize content delivery to and / or playback by the population of streaming clients.

[0159] As a further application or use case, the proposed technique could be used to provide a feedback loop for content edge processing. For example, if the content encoder is located at the edge of the network, an estimated value of audio quality (e.g., a MUSHRA score) could be used to fine-tune the bitrate ladder used to deliver the audio content to the client.

[0160] An example of a framework including such a feedback loop is schematically shown in Figure 10. The content server 1050 (content origin) provides (audio) content to the encoding and packaging coordination engine 1090 (encoding and packaging engine), which encodes and packages the content according to one or more sets of rules, and provides the encoded and packaged content to a point of presence (PoP) in the CDN 1030 (or multiple CDNs). The rules used by the encoding and packaging engine 1090 may relate, for example, to maximizing a value function and / or minimizing a cost function. For example, the one or more rules may relate to minimizing the number of levels in a bitrate ladder while optimizing mean (worst-case) performance and / or determining the optimal bitrate to maximize mean (worst-case) performance.

[0161] CDN 1030 provides (audio) content to a population 1010 of streaming clients 1015, and the clients provide playback-related information 1070 to a quality evaluation service 1060. The quality evaluation service 1060 determines the playback performance (e.g., average playback performance, worst-case playback performance, etc.) for the population 1010 of streaming clients 1015 using the techniques described above, and provides the determined playback performance instructions to the encoding and packaging engine 1090.

[0162] Following the techniques described above, playback performance may be related to the aforementioned estimates of audio quality, or the quantities derived therefrom.

[0163] Therefore, generally, step S240 of the method 200 described above may include, or relate to, outputting a determined estimate of the audio quality of the test audio signal (for example, as playback performance) to a network node (for example, the encoding and packaging engine 1090 described above) for performing the encoding and / or packaging of audio content.

[0164] Based on the playback performance 1065, the encoding and packaging engine 1090 in the example of Figure 11 can then optimize the encoding and / or packaging of the content.

[0165] For example, in a framework like the one shown in the example in Figure 10, one or more of the steps S1110 to S1130 of method 1100 shown in the flowchart in Figure 11 may be performed.

[0166] Step S1110 This relates to, or includes, optimizing encoding and / or packaging based on an estimate of the audio quality of a test audio signal.

[0167] Step S1120 This relates to, or includes, determining the optimal number of quality levels in a bitrate ladder for delivery by a content delivery network, based on an estimate of the audio quality of a test audio signal. This may involve, for example, inputting the estimate of the audio quality of the test audio signal into a utility function (e.g., a value function and / or cost function) used to determine the optimal number of quality levels.

[0168] Step S1130 This relates to, or includes, determining the configuration and / or set of encoding tools based on an estimate of the audio quality of the test audio signal.

[0169] Apparatus for carrying out the method according to the present invention Finally, while the above references may be made primarily to the methods described herein, this disclosure also relates to apparatus (e.g., computer-implemented apparatus) for performing the methods and techniques described throughout this disclosure. An example of such apparatus 2100, described in more detail below, is schematically shown in Figure 21. Such apparatus may implement, for example, a quality evaluation service 150 or an (instrumentation) streaming client 10. The apparatus (e.g., its processor 2110) may, among other things, receive suitable input data (e.g., playback-related information or instructions for audio content processed by the streaming client), depending on the use case and / or implementation. The device 2100 (for example, its processor 2110) may, depending on the use case and / or implementation, be adapted to perform the methods / techniques described throughout this disclosure (for example, method 200 in Figure 2, method 400 in Figure 4A, method 450 in Figure 4B, method 700 in Figure 7, and / or method 1100 in Figure 11) and generate the corresponding output data 1240 (for example, an estimate of audio quality).

[0170] This disclosure also relates to corresponding computer programs and computer-readable storage media.

[0171] Machine listener One non-exclusive example of implementing the aforementioned quality evaluation service, or the algorithms used by the quality evaluation service, is a so-called machine listener (e.g., a generating machine listener), as described below.

[0172] Broadly speaking, a machine listener (e.g., a generative machine listener) in an embodiment of the present disclosure is a neural network (e.g., a deep neural network) trained to evaluate audio by comparing it to a relevant reference signal and providing evaluation results as a probability distribution of predicted listener scores in a currency of subjective listening tests, such as Multi-Stimulus Tests with Hidden Reference and Anchor (MUSHRA).

[0173] In general, listener scores achieved in tests such as the MUSHRA test can be predicted by a system that takes the test signal and a reference signal as inputs. An example of such a system, including a neural network, is shown in Patent Document 1, which is included herein by reference in its entirety.

[0174] For a given input signal pair, there is typically some variability in the listener scores perceived by different listeners. Capturing this aspect of the data has proven valuable for automated estimation of the quality of experience in entertainment delivery systems. In actual subjective MUSHRA trials, the mean and standard deviation of listener scores obtained from different listeners can be calculated, and then, given the number of listeners and a statistical model, the standard deviation can be converted into a confidence interval.

[0175] The inventors found it difficult to train a neural network to directly output both the mean and confidence interval. One feasible alternative for quantifying predictive variability could be to use a bootstrapping approach, which involves training multiple models on a randomly sampled subset of the data with the mean score as the objective. Each of the trained models then introduces slightly different predictions. The variability of these predictions then quantifies the confidence level. However, the bootstrapping method has two drawbacks. First, using the bootstrapping method implies high complexity and requires as many models as there are listeners being simulated. Second, there is a risk of modeling predictive variability rather than the listener score distribution.

[0176] Another feasible alternative for quantifying predictive variability could be to consider separate modeling of individual listener bias (see Non-Patent Documents 1 and 2) or bias and inconsistency (see Non-Patent Documents 3 and 4) across the signal under test. The primary application of the latter method is pre-screening listeners with outlier behavior. A common drawback of these methods is the need to track listener characteristics within the dataset.

[0177] This disclosure aims to provide an improved technique for quantifying the predictive variability of subjective listening tests. In the application stage (i.e., in inference), the trained model (e.g., a generative model) provided by this disclosure provides a distribution of scores from which the mean score, standard deviation, and / or confidence interval (CI) for any number of listeners can be easily extracted.

[0178] Unlike existing methods that use the mean subjective score as the target for training during the training stage, the model according to this disclosure utilizes individual listener scores. This has been shown to simplify preprocessing and allows the dataset to influence training in a manner directly proportional to the human effort for listening test data with varying numbers of listeners. Furthermore, the maximum likelihood principle can be used for parameter estimation.

[0179] The technique according to the present invention has been found to have the following advantages: The trained model (e.g., a generative model) achieves performance similar to that of a typical non-generative model when predicting the mean, but can also predict confidence intervals. Furthermore, the trained model is more robust to conditions not typically seen in listening tests.

[0180] Figure 12 shows a comparison between a conventional model (e.g., a DNN) for predicting an average listening score (e.g., a MUSHRA score) and a model according to embodiments of the present disclosure (e.g., a DNN implementing a generative machine listener). The conventional model shown on the left (a non-generative approach) is given ref-coded audio (or its features, such as a gammatone spectrogram) and predicts an average subjective listening score (e.g., a MUSHRA score). On the other hand, the generative machine listener model shown on the right provides a distribution of listening scores (e.g., MUSHRA scores). In training, the present disclosure proposes to utilize individual listening scores. While not intended to be limiting, the architecture of the DNN implementing a generative machine listener may be as described in Patent Document 1, with the difference being that the output stage is configured to provide two or more outputs suitable for representing, for example, a distribution of listening scores (e.g., MUSHRA scores).

[0181] Description of Exemplary Embodiments Given the original signal x and the signal under test y, the generative listener model (or the DNN implementing it) provides an indication of the probability distribution of the (subjective) listening score (e.g., MUSHRA score) s for y, as the parameterized probability density p θ (s│x,y) (1) is given as.

[0182] The parameters θ of the model are trained by the maximum likelihood principle. Then, by sampling the model N times, a listening test with N listeners can be simulated. The negative log likelihood (NLL) loss used for training the model parameters θ with respect to the listener score values s in the dataset can be -log p θ (s│x,y).

[0183] Generally, in the training stage, the input is given by training data items, each representing a different value of the listening score s. The loss function depends on the indication of the listening score s. Each training data item may further be indicated as a representation of the audio signal (the signal under test y) and a reference audio signal for the audio signal (the original signal x). Here, the representation of the audio signal y and the representation of the reference audio signal x may relate to, for example, a gammatone spectrogram. Each training data item can be obtained by performing a standardized listening test (e.g., a MUSHRA test) on the test signal y and the corresponding reference signal x and giving a listening score s. Performing such a test multiple times, for example, with different listeners, will result in multiple training data items tentatively denoted as (s,y,x). The test signal y and the corresponding reference signal x may be obtained, for example, from a suitable audio library. Here, and for the remainder of this disclosure, it will be understood that the reference signal corresponds to an unencoded signal. The test audio signal corresponds to an encoded audio signal (for example, a signal obtained after encoding, decoding, and time-aligning with a reference signal to compensate for encoding delays if necessary).

[0184] If the representations of the test signal y and the reference signal x are in terms of gammatone spectrograms (typically L, R, M, and S spectrograms), then each actual sound signal may yield four training data items (e.g., one for each of the L, R, M, and S spectrograms).

[0185] An example of Method 1300 for configuring (e.g., training) a DNN to estimate subjective listening score indications for audio signals is shown in the flowchart of Figure 13. Method 1300 includes steps S1310 and S1320.

[0186] It is understood that a DNN implements the model under consideration (for example, a generative model). For example, a DNN could implement the aforementioned generative machine listener.

[0187] Furthermore, the listening scores estimated or predicted by the DNN may be scores that follow a predefined (e.g., standardized) listening test. The listening test may apply predefined test metrics and / or test scenarios. An example of such a listening test is the MUSHRA listening test.

[0188] Step S1310 In this configuration, an output stage of a DNN is provided for generating instructions for listening scores.

[0189] Step S1320 In this scenario, the DNN is trained in at least one of the multiple training epochs.

[0190] Method 1400, shown in the flowchart of Figure 14, is an example of a possible implementation in which, in step S1320, the DNN is trained in one of several training epochs. Method 1400 includes steps S1410 to S1440.

[0191] Step S1410 Next, one or more training data items are entered. For example, a mini-batch of training data items (e.g., eight training data items) may be entered. As mentioned above, each training data item represents a corresponding value of the listening score s. Also as mentioned above, each training data item may represent a representation of the audio signal (test signal y) and a reference audio signal (original signal x) for the audio signal.

[0192] Step S1420 Then, each instruction in the listening score is determined based on one or more of the training data items. These instructions may be determined, for example, based on the representation of the audio signal and the representation of the reference audio signal.

[0193] In some embodiments, the instruction for the listening score may also be related to the probability distribution of the listening score (e.g., a probability density function), and the output stage is adapted to generate this probability distribution (e.g., p by equation (1)). θ (s│x,y)) can emulate listening scores obtained from multiple (independent) listening tests for that audio signal. Furthermore, the probability distribution may be parameterized by two or more parameters of the probability distribution. Examples of possible parameterization of the probability distribution will be described later.

[0194] If the indication of the listening score is related to a probability distribution of listening scores, determining each indication of the listening score based on one or more training data items may include determining each parameter of the probability distribution based on one or more training data items. Here, determining the parameters of the probability distribution may be based at least in part on the subjective listening score values ​​included in the training data items. Furthermore, determining the parameters of the probability distribution may also be based on the current state of the DNN, for example, the current values ​​of the DNN's internal parameters.

[0195] Step S1430 Then, by evaluating the loss function, the respective loss values ​​for one or more training data items are determined. Here, the loss function depends on the indication of the listening score.

[0196] In some embodiments, if the indication of the listening score is related to the probability distribution of the listening score, the loss function may depend on the parameters of the distribution. Examples of loss functions will be discussed later.

[0197] Step S1440 Then, one or more internal parameters of the DNN are adjusted based on the loss value determined, for example, by using well-known regression and backpropagation techniques. The internal parameters of the DNN may also be model parameters, such as coefficients for multiple layers of the DNN (e.g., filter coefficients).

[0198] If multiple training data items are entered for each training epoch, the internal parameters may be adjusted based on an aggregation of loss values ​​for the training data items, such as the mean or average of the loss values ​​for the training data items.

[0199] As mentioned above, training a DNN via method 1400, for example, may be based on the maximum likelihood principle. Therefore, the loss function used for training (e.g., the loss function evaluated in step S1430 of method 1400) may be related to the negative log-likelihood (NLL) loss. Again, as mentioned above, the negative log-likelihood loss is L NLL = -log p θ (s|x,y) (2) It may be given by p. Here, p θ (s|x,y) is the probability density function (an example of a probability distribution) for the test score s given a representation of the audio signal y and a reference audio signal x for the audio signal y, where θ represents the internal parameter of the DNN.

[0200] probability density function p θ The first non-restrictive example of (s|x,y) is given by mean μ and variance σ. 2 It is a Gaussian distribution parameterized by [a certain factor]. In that case, the NLL loss is, for example,

number

[0201] Thus, the probability distribution in step S1420 of method 1400 is given by mean μ and variance σ. 2 It may also relate to a Gaussian distribution parameterized by . Then the loss function L Gauss teeth,

number

[0202] probability density function p θ A second non-restrictive example of (s|x,y) is that of a logistic distribution parameterized by mean μ and scale a. The NLL loss in this case is, for example,

number

[0203] Thus, the probability distribution in step S1420 of method 1400 may be related to a logistic distribution parameterized by mean μ and scale a. In that case, the loss function L logistic teeth,

number

[0204] Models with more than two parameters, such as Gaussian or logistic mixtures, or even categorical distributions, may have the ability to model multimodal listener score distributions. However, they have the potential drawback of requiring more data to be successfully trained.

[0205] In inference, an estimate of the subjective listening score indication for an audio signal can be determined using a well-trained DNN, for example, a DNN trained as described above. As described above, the listening score is assumed to be a score according to a predefined listening test. Furthermore, the DNN is generally assumed to comprise an input stage for receiving a representation of the audio signal and a representation of a reference audio signal for the audio signal, a plurality of layers for performing processing based on the representation of the audio signal and the representation of the reference audio signal, and an output stage coupled to the last of the plurality of layers for generating the listening score indication. Here, the processing by the plurality of layers may further depend on the current state of the DNN, for example, the current values ​​of the DNN's internal parameters.

[0206] An example of the corresponding method 1500 using this DNN is shown in the flowchart of Figure 15. Method 1500 includes steps S1510 and S1520.

[0207] Step S1510 The representation of the audio signal and the representation of the reference audio signal are then input to the input stage of the DNN.

[0208] Step S1520 Therefore, the expression of the listening score instruction is determined based on the output of the DNN's output stage.

[0209] As described above, the indication of the listening score may be related to a probability distribution of the listening score, and the output stage of the DNN is adapted to generate a probability distribution of the listening score. This probability distribution can be considered to emulate the listening score obtained by multiple listening tests on an audio signal. Furthermore, the probability distribution may be parameterized by two or more parameters of the probability distribution. Thus, the expression of the indication of the listening score determined in step S1520 may be related, for example, to the parameters of the probability distribution.

[0210] Furthermore, by making the output of the DNN's output stage available, the representation of the probability distribution can be determined, for example, by determining at least one of the mean, standard deviation, and confidence interval from the output of the output stage.

[0211] For example, the confidence interval can be determined based on the output of the output stage and the number of listeners in the listening test being emulated. Referring to the above example of Gaussian parameterization of a probability distribution, once the parameter σ is determined, a 95% confidence interval (CI) can be calculated. 95 )but CI 95 = 1.96(σ / √N) (5) It can be calculated as follows.

[0212] It is understood that a similar decision can also be applied to the parameterization of probability distributions, such as the logistic distribution, based on scale a.

[0213] In addition to the methods described above, this disclosure also relates to a DNN for estimating a subjective listening score indication for an audio signal. Here again, the listening score may be a score according to a predefined listening test, such as the MUSHRA test. Such a DNN may comprise an input stage for receiving a representation of the audio signal (e.g., one or more gammatone spectrograms) and a representation of a reference audio signal for the audio signal (e.g., one or more gammatone spectrograms), a plurality of layers for performing processing based on the representation of the audio signal and the representation of the reference audio signal, and an output stage for generating a listening score indication. It is understood that the first of the plurality of layers is coupled to the input stage, and the last of the plurality of layers is coupled to the output stage. The processing by the plurality of layers may further depend on the current state of the DNN, for example, the current values ​​of the DNN's internal parameters.

[0214] It is understood that DNNs can be implemented by any suitable computing system, such as the device shown in Figure 21.

[0215] Furthermore, the DNN may be constructed (e.g., trained) by training the DNN according to the method 1400 described above.

[0216] In particular, the DNN may be trained by performing the following steps in a training epoch among multiple training epochs: inputting one or more training data items, each representing a different value of the listening score; determining each indication of the listening score based on the one or more training data items; determining each loss value for the one or more training data items by evaluating a loss function, wherein the loss function depends on the indication of the listening score; and adjusting one or more internal parameters of the DNN based on the determined loss values.

[0217] As described above, the instruction for the listening score may also be related to the probability distribution of the listening score, and the output stage of the DNN is adapted to generate the probability distribution of the listening score. This probability distribution may emulate listening scores obtained from multiple listening tests on an audio signal, and may be parameterized by two or more parameters of the probability distribution. For example, the probability distribution may have a mean μ and variance σ as described above. 2 It may relate to a Gaussian distribution parameterized by μ, or to a logistic distribution parameterized by mean μ and scale a.

[0218] If the indication of the listening score is related to the probability distribution of the listening score, determining each indication of the listening score based on one or more training data items may include determining each parameter of the probability distribution based on one or more training data items, for example, at least in part on the subjective listening score values. Furthermore, in this case, the loss function depends on the parameters of the distribution. It is also understood that determining the parameters of the probability distribution may be based on the current state of the DNN, for example, the current values ​​of the DNN's internal parameters.

[0219] Simulation results for an exemplary test set Two stereo listening tests were considered as the test set. One listening test tested a low-bitrate codec, and the other tested a high-bitrate codec. A strategy was devised to select the best model from several models trained over epochs.

[0220] When selecting a model, the following factors can be considered. • Stability of the training process. For example, generally, models trained with a logistic distribution exhibit smoother training and validation loss decay than models trained with a Gaussian distribution under the same settings. However, Gaussians can still be a promising option when fine-tuning the training process using techniques such as gradient clipping. • The Pearson Correlation Coefficient (PCC) between the predicted average MUSHRA score and the actual average MUSHRA listening test score. A higher PCC (closer to 1) is preferable. • NLL loss of the training and validation sets. For example, a model trained on a Gaussian distribution will exhibit a much smaller NLL loss than a model trained on a logistic distribution.

[0221] Taking the above aspects into consideration, we selected several models (from different epochs, i.e., different stages of training; for either Gaussian or logistic models) that had the highest PCC, lower training NLL loss, and slightly lower validation loss on the validation set. For models with similar PCC scores, those with lower training NLL loss (typically models generated in later epochs) were retained, along with slightly lower validation NLL loss. The reason for the latter is that we have found that the model with the lowest validation loss does not necessarily perform best when predicting confidence intervals on the test set.

[0222] Figure 16 is a plot showing examples of mean NLL loss in two listening tests using Gaussian and logistic distributions. Overall, the logistic distribution shows higher NLL loss than the Gaussian model on the test set. However, lower loss does not necessarily mean a better model (for example, visually, the logistic model shows a closer fit to the correct answer).

[0223] For the plots shown in Figures 17 to 20, each excerpt has a reference, a 3.5 kHz anchor, and a 7 kHz anchor, followed by a different encoded representation.

[0224] Figure 17 is a plot illustrating exemplary results from a stereo low-bitrate test. Specifically, the plot shows the accuracy of predicting the mean MUSHRA score using generative machine listeners trained on logistic and Gaussian distributions for different classes of audio (three utterances, music, and mixed excerpts).

[0225] Figure 18 is another plot illustrating exemplary results from a stereo low-bitrate test. Specifically, the plot shows the accuracy of predicting CI using generative machine listeners trained on logistic and Gaussian distributions for 44 listeners and different classes of audio (three utterances, music, and mixed excerpts).

[0226] Figure 19 is a plot illustrating exemplary results from a stereo high-bitrate test. Specifically, the plot shows the accuracy of predicting the mean MUSHRA score using generative machine listeners trained on logistic and Gaussian distributions for different classes of audio (three utterances, music, and mixed excerpts).

[0227] Finally, Figure 20 is a plot showing exemplary results from a stereo high-bitrate test. Specifically, the plot shows the accuracy of predicting CI using a generative machine listener trained on logistic and Gaussian distributions for 28 listeners and different classes of audio (three utterances, music, and mixed excerpts).

[0228] Pearson correlation (PC) evaluates the linear relationship between two continuous variables. For the examples shown in the plots in Figures 17-20, the trend for predicting CI is closer to the ground truth (i.e., CI from the listening test) when using a model trained with a logistic distribution (PC=0.8190) than with a Gaussian distribution (PC=0.6733). For high bitrates, both the logistic distribution (PC=0.6461) and the Gaussian distribution (PC=0.6700) are equivalent.

[0229] To predict the mean, both models perform equally well in the above example. For low bitrate tests, the model trained with a logistic distribution has a PC of 0.9172, while the Gaussian distribution has a PC of 0.9183. For high bitrate tests, the model trained with a logistic distribution has a PC of 0.9316, while the Gaussian distribution has a PC of 0.9375.

[0230] Spearman correlation (SC) evaluates monotonic relationships. The Spearman correlation coefficient is based on ranked values ​​for each variable, rather than the raw data. This measures rank preservation. For low-bitrate tests, in the example above, the model trained on a logistic distribution has an SC of 0.8991, while the Gaussian distribution has an SC of 0.8716. For high-bitrate tests, the model trained on a logistic distribution has an SC of 0.9297, while the Gaussian distribution has an SC of 0.9360.

[0231] It should be noted that in subjective scores, bitrates associated with higher quality, such as references, low-pass anchors, and generally higher quality, are rated with lower CIs, and bitrates associated with lower quality are rated with higher CIs. Unlike the aforementioned bootstrapping approach, where higher CIs are associated with higher density data points in the training set (and vice versa), the technique presented in this disclosure models the variability of listener scores.

[0232] Apparatus for carrying out the method according to the present invention Finally, while the above references may be made primarily to the methods described herein, this disclosure also relates to apparatus (e.g., computer-implemented apparatus) for carrying out the methods and techniques described throughout this disclosure. An example of such apparatus 2100 is schematically shown in Figure 21. Such apparatus 2100 may, for example, implement the deep neural network (machine listener) described above. Apparatus 2100 comprises a processor 2110 and a memory 2120 coupled to the processor 2110. The memory 2120 may store instructions for the processor 2110. The processor 2110 may also, depending on the use case and / or implementation, receive appropriate input data (e.g., appropriate training data in the training stage, or appropriate test and reference audio signals in inference). The processor 2110 may be adapted, depending on the use case and / or implementation, to perform the methods / techniques described throughout this disclosure (e.g., method 1300 in Figure 13, method 1400 in Figure 14, or method 1500 in Figure 15) and generate the corresponding output data 1240 (e.g., listening score instructions). For example, the device 2100 may implement the method for training the DNN described above, or it may implement the (trained) DNN described above.

[0233] This disclosure also relates to corresponding computer programs and computer-readable storage media.

[0234] interpretation Aspects of the systems described herein may be implemented in a suitable computer-based audio processing network environment (e.g., a server or cloud environment) for processing digital or digitized audio files. Part of the adaptive audio system may include one or more networks containing any desired number of individual machines, including one or more routers (not shown) that work to buffer and route data transmitted between computers. Such networks may be built on a variety of different network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0235] One or more components, blocks, processes, or other functional components may be implemented through computer programs that control the execution of the system's processor-based computing devices. It should also be noted that the various functions disclosed herein may be described using any number of combinations of hardware and firmware, and / or as data and / or instructions embodied in various machine-readable or computer-readable media with respect to their behavior, register transfers, logical components, and / or other characteristics. Computer-readable media in which such formatted data and / or instructions may be embodied include, but are not limited to, various forms of physical (non-transient) non-volatile storage media, such as optical, magnetic, or semiconductor storage media.

[0236] Specifically, embodiments may include hardware, software, and electronic components or modules, and it should be understood that for the purposes of discussion, these may be illustrated and described as if most of the components were implemented only in hardware. However, those skilled in the art will recognize that based on reading this detailed description, in at least one embodiment, the electronic-based aspects may be implemented in software executable by one or more electronic processors such as a microprocessor and / or an application specific integrated circuit (“ASIC”) (e.g., stored on a non-transitory computer-readable medium). Thus, note that multiple hardware and software-based devices, as well as multiple different structural components, may be utilized to implement embodiments. For example, the systems, services, clients, nodes, etc. described in the context of FIGS. 1, 3A, 3B, 5, 8, 9, 10, 12, and / or 21 above may include or be implemented by one or more electronic processors, one or more computer-readable media modules, one or more input / output interfaces, and various connections (e.g., a system bus) connecting the various components.

[0237] One or more implementations have been described by way of example with respect to specific embodiments, but it should be understood that one or more implementations are not limited to the disclosed embodiments. On the contrary, as will be apparent to those skilled in the art, various modifications and similar configurations are intended to be covered. Thus, the appended claims should be given the broadest interpretation so as to encompass all such modifications and similar configurations.

[0238] Also, it should be understood that the expressions and terms used in this specification are for the purpose of explanation and should not be regarded as limiting. The use of "including", "comprising", or "having" and their variations is intended to include the listed items and their equivalents as well as additional items. Unless specifically specified or limited, the terms "attached", "connected", "supported", and "coupled", and their variations are used in a broad sense and include both direct and indirect attachment, connection, support, and coupling.

[0239] Bulleted Exemplary Embodiments Various aspects and implementations of the present disclosure can also be understood from the following itemized example embodiments (enumerated example embodiment, EEE) which are not the claims.

[0240] 〔EEE1〕 A method for evaluating the playback performance of adaptive streaming for a streaming client, the method using an objective quality evaluation algorithm that emulates an intrusive quality test, the method being implemented on a network node different from the client, the method comprising the steps of receiving playback-related metadata from the client; reconstructing a description of the test signal as required by the objective quality evaluation algorithm using the metadata; reconstructing a description of the reference signal as required by the objective quality evaluation algorithm; calculating the performance estimate and distributing the performance estimate to other network nodes other than the node on which the method operates.

[0241] 〔EEE2〕 The method according to EEE1, wherein the objective quality evaluation algorithm uses a set of pre-trained models and performs model selection based on the metadata received from the client.

[0242] 〔EEE3〕 The method of EEE1 or EEE2, wherein the metadata received from the client includes information that facilitates the reconstruction of the description of the test signal required by the objective quality evaluation tool at a network node implementing the method, such as: segment name and bitrate; and / or a representation of a set of spectrograms calculated based on the segment content (as required by the objective quality evaluation algorithm).

[0243] [EEE4] The method according to EEE3, wherein the metadata received from the client includes information about the playback device and / or playback conditions, for example: an indication of the playback device (headphones, soundbar, or discrete speakers); characteristics of the playback conditions (e.g., signal-to-noise ratio); and / or other information.

[0244] [EEE5] Any of the aforementioned EEE methods, wherein the estimates provided by the method are used to compare at least two different groups of clients.

[0245] [EEE6] Any of the aforementioned methods of the EEE, wherein the performance estimate calculated by the said method is provided to a service that optimizes the encoding / packaging of the content.

[0246] [EEE7] The method according to EEE6, wherein the performance estimate is used as input to a utility function used to determine the optimal number of quality levels in the bitrate ladder distributed across the points of presence (PoPs) of the CDN.

[0247] [EEE8] The method according to EEE6, wherein the performance estimate is used to determine the tuning of the content encoder (e.g., configuration, set of encoding tools).

[0248] [EEE-A1] A method for constructing a deep neural network (DNN) for estimating subjective listening score indications for audio signals, wherein the listening score is a score according to a predefined listening test, and the method is: A step of providing an output stage of the DNN that generates the instructions for the listening score; The process includes the step of training the DNN, and training the DNN involves: In one of several training epochs, the step of inputting one or more training data items that represent each of the values ​​of the aforementioned listening scores; A step of determining the respective indications of the listening score based on one or more of the training data items; A step of determining the respective loss value for one or more training data items by evaluating the loss function, wherein the loss function depends on the indication of the listening score; This involves performing the step of adjusting one or more internal parameters of the DNN based on the determined loss value, method.

[0249] [EEE-A2] The instruction for the listening score relates to a probability distribution of the listening score, the output stage is adapted to generate the probability distribution of the listening score, the probability distribution emulates listening scores obtained by multiple listening tests of the audio signal, and the probability distribution is parameterized by two or more parameters of the probability distribution; Determining each indication of the listening score based on one or more training data items includes determining each parameter of the probability distribution based on one or more training data items; The loss function depends on the parameters of the distribution. Method described in EEE-A1.

[0250] 〔EEE-A3〕 Training the DNN is the method described in EEE-A1 or 2 based on the maximum likelihood principle.

[0251] 〔EEE-A4〕 The loss function is the method described in any one of EEE-A1 to 3 related to the negative log-likelihood (NLL) loss.

[0252] 〔EEE-A5〕 The negative log-likelihood loss is L NLL =-log p θ (s|x,y), where p θ (s|x,y) is the probability distribution of the test score s when the representation of the audio signal y and the representation of the reference audio signal x for the audio signal y are given, and θ represents the internal parameters of the DNN. The method described in EEE-A4 when citing EEE-A2.

[0253] 〔EEE-A6〕 The probability distribution is related to a Gaussian distribution parameterized by a mean μ and a variance σ 2 , and the loss function L Gauss is

Equation

[0254] 〔EEE-A7〕 The probability distribution is related to a logistic distribution parameterized by a mean μ and a scale a, and the loss function L logistic is

Equation

[0255] [EEE-A8] The method according to any one of EEE-A1 to 7, wherein the training data item further indicates a representation of the audio signal and a representation of a reference audio signal for the audio signal.

[0256] [EEE-A9] The representation of the audio signal and the representation of the reference audio signal are as described in EEE-A8, relating to a gammatone spectrogram.

[0257] [EEE-A10] The aforementioned predefined listening test is a multiple-stimulus listening test with hidden references and anchors (MUSHRA) as described in any one of EEE-A1 through 9.

[0258] [EEE-A11] The aforementioned DNN is the method described in any one of EEE-A1 to 10 for implementing the generative model.

[0259] [EEE-A12] A method for estimating a subjective listening score for an audio signal using a deep neural network (DNN), wherein the listening score is a score obtained from a predefined listening test. The aforementioned DNN is: An input stage for receiving the representation of the audio signal and a representation of a reference audio signal for the audio signal; Multiple layers for performing processing based on the representation of the audio signal and the representation of the reference audio signal; It has an output stage for generating the instructions for the listening score, The method in question is: A step of inputting the representation of the aforementioned audio signal and the representation of the aforementioned reference audio signal to the input stage; The step includes determining the representation of the instruction in the listening score based on the output of the output stage, method.

[0260] [EEE-A13] The method according to EEE-A12, wherein the index of the listening score relates to a probability distribution of the listening score, the output stage of the DNN is adapted to generate the probability distribution of the listening score, the probability distribution emulates a listening score obtained by multiple listening tests of the audio signal, and the probability distribution is parameterized by two or more parameters of the probability distribution.

[0261] [EEE-A14] The method according to EEE-A13, wherein determining the representation of the probability distribution includes determining at least one of the mean, standard deviation, and confidence interval from the output of the output stage.

[0262] [EEE-A15] The method according to EEE-A14, wherein the confidence interval is determined based on the output of the output stage and the number of listeners in the listening test to be emulated.

[0263] [EEE-A16] The aforementioned probability distribution has a mean μ and a variance σ. 2 The method described in any one of EEE-A13 to EEE-A15 relating to a Gaussian distribution parameterized by or a logistic distribution parameterized by mean μ and scale a.

[0264] [EEE-A17] The representation of the audio signal and the representation of the reference audio signal are as described in any one of EEE-A13 to EEE-A16 relating to a gammatone spectrogram.

[0265] [EEE-A18] The aforementioned predefined listening test is a multiple-stimulus listening test with hidden references and anchors (MUSHRA) as described in any one of EEE-A13 to 17.

[0266] [EEE-A19] A deep neural network (DNN) for estimating subjective listening score indications for audio signals, wherein the DNN is: An input stage for receiving the representation of the audio signal and a representation of a reference audio signal for the audio signal; Multiple layers for performing processing based on the representation of the audio signal and the representation of the reference audio signal; The system includes an output stage for generating the instructions for the listening score, DNN.

[0267] [EEE-A20] The DNN may be comprised of training the DNN, the training comprising: in one of a plurality of training epochs: inputting one or more training data items representing each of the listening scores; A step of determining the respective indications of the listening score based on one or more of the training data items; A step of determining the respective loss value for one or more training data items by evaluating the loss function, wherein the loss function depends on the indication of the listening score; This involves performing the step of adjusting one or more internal parameters of the DNN based on the determined loss value, DNN as described in EEE-A19.

[0268] [EEE-A21] The DNN according to EEE-A19 or 20, wherein the instruction for the listening score relates to a probability distribution of the listening score, the output stage of the DNN is adapted to generate the probability distribution of the listening score, the probability distribution emulates listening scores obtained by a plurality of listening tests for the audio signal, and the probability distribution is parameterized by two or more parameters of the probability distribution.

[0269] [EEE-A22] Determining each indication of the listening score based on one or more training data items includes determining each parameter of the probability distribution based on one or more training data items; The loss function depends on the parameters of the distribution. The DNN described in EEE-A21 when citing EEE-A20.

[0270] [EEE-A23] The aforementioned probability distribution has a mean μ and a variance σ. 2 A DNN as described in EEE-A21 or 22, relating to a Gaussian distribution parameterized by, or a logistic distribution parameterized by mean μ and scale a.

[0271] [EEE-A24] The representation of the audio signal and the representation of the reference audio signal are DNNs as described in any one of EEE-A19 to 23 relating to gammatone spectrograms.

[0272] [EEE-A25] The aforementioned predefined listening test is a multiple-stimulus listening test with hidden references and anchors (MUSHRA) as described in any one of EEE-A19 to 24.

[0273] [EEE-A26] An apparatus comprising a processor and a memory coupled to the processor and storing instructions for the processor, wherein the processor is adapted to perform the method described in any one of EEE-A1 to 18.

[0274] [EEE-A27] A device comprising a processor and a memory coupled to the processor and storing instructions for the processor, wherein the processor is adapted to implement a DNN as described in any one of EEE-A19 to EEE-A25.

[0275] [EEE-A28] A program that, when executed by a processor, includes instructions that cause the processor to perform the method described in any one of EEE-A1 to 18.

[0276] [EEE-A29] A program that, when executed by a processor, includes instructions that cause the processor to implement a DNN described in any one of EEE-A19 to 25.

[0277] [EEE-A30] A computer-readable storage medium storing a program as described in EEE-A28 or 29.

[0278] [EEE-B1] A method for evaluating playback performance in an adaptive streaming environment, wherein the method is: The stage of obtaining playback-related information from the streaming client; A step of estimating the representation of a test audio signal based on the playback-related information, wherein the test audio signal is an audio signal played back by the streaming client; A step of determining an estimated value of the audio quality of the test audio signal based on an estimated representation of the test audio signal using an audio quality evaluation algorithm. Methods that include...

[0279] [EEE-B2] The step further includes generating a representation of a reference audio signal for the aforementioned test audio signal, The method described in EEE-B1. [EEE-B3] The process further includes obtaining instructions for audio content processed by the streaming client from the streaming client, The method described in EEE-B1 or 2.

[0280] [EEE-B4] Estimating the representation of the test audio signal is the method of EEE-B3, further based on the instructions of the audio content.

[0281] [EEE-B5] Generating the representation of the reference audio signal is the method according to EEE-B3 or 4, when referring to EEE-B2, based on the instructions of the audio content.

[0282] [EEE-B6] The playback-related information includes bitrate information indicating the bitrate of the audio signal played back by the streaming client. Estimating the representation of the test audio signal is based on the bitrate information, The method described in any one of the EEE-B1 to 5.

[0283] [EEE-B7] The aforementioned audio quality evaluation algorithm uses a set of pre-trained models for audio quality evaluation, Generating the estimated audio quality includes selecting a pre-trained model from the set of pre-trained models based on the playback-related information. The method described in any one of the EEE-B1 to 6.

[0284] [EEE-B8] The aforementioned playback-related information includes information related to the playback device associated with the streaming client, The pre-trained model is selected based on information related to the playback device. Methods for EEE-B7.

[0285] [EEE-B9] The audio quality evaluation algorithm is implemented by a deep neural network (DNN) for estimating an indication of a subjective listening score for the representation of a test audio signal as an estimate of the audio quality, wherein the listening score is a score from a predefined listening test, and the DNN is: An input stage for receiving a representation of the test audio signal and a representation of a reference audio signal for the test audio signal; Multiple layers for performing processing based on the representation of the test audio signal and the representation of the reference audio signal; Includes an output stage for generating the instructions for the listening score, The method described in any one of the EEE-B1 to 8.

[0286] [EEE-B10] The aforementioned DNN operates in one of several training epochs: The steps include: inputting one or more training data items that represent the respective values ​​of the aforementioned listening scores; A step of determining the respective indications of the listening score based on one or more of the training data items; A step of determining the respective loss value for one or more training data items by evaluating the loss function, wherein the loss function depends on the indication of the listening score; By performing the step of adjusting one or more internal parameters of the DNN based on the determined loss value, The method described in EEE-B9, which is constructed by training the aforementioned DNN.

[0287] [EEE-B11] The method is the method according to any one of EEE-B1 to 10, performed on a network node different from the streaming client.

[0288] [EEE-B12] The representation of the estimated value of the test audio signal is as described in any one of EEE-B1 to 11, relating to one or more gammatone spectrograms.

[0289] [EEE-B13] The representation of the reference audio signal is as described in any one of EEE-B2 or, when referring to EEE-B2, EEE-B3 to 12, relating to one or more gammatone spectrograms.

[0290] [EEE-B14] The method according to any one of EEE-B1 to 13, further comprising outputting an estimate of the audio quality of the test audio signal to a network node different from the network node associated with the streaming client.

[0291] [EEE-B15] The estimated value of the audio quality of the test audio signal is output to a network node to perform encoding and / or packaging of the audio content. The method further includes the step of optimizing the encoding and / or packaging based on the estimated value of the audio quality of the test audio signal. The method described in any one of EEE-B1 to 14.

[0292] [EEE-B16] The method according to EEE-B15, further comprising determining an optimal number of quality levels in a bitrate ladder for distribution by a content delivery network based on the estimated value of the audio quality of the test audio signal.

[0293] [EEE-B17] The method according to EEE-B15 or 16, further comprising determining the configuration and / or set of encoding tools based on the estimated value of the audio quality of the test audio signal.

[0294] [EEE-B18] The steps include determining an estimate of the audio quality of a test audio signal for each streaming client in a group of multiple streaming clients; A step of comparing estimated audio quality values ​​determined for the aforementioned group of streaming clients. The method described in any one of EEE-B1 to 17, further including the method described in any one of EEE-B1 to 17.

[0295] [EEE-B19] A method for providing playback-related information in a streaming client that processes audio content in an adaptive streaming environment, the method being: To analyze the playback buffer associated with the streaming client in order to determine bitrate information indicating the bitrate of the audio signal segment played back by the streaming client; Analyzing the manifest information associated with the aforementioned audio content; and By analyzing one or more of the characteristics of the playback device associated with the aforementioned streaming client The stage of generating the aforementioned playback-related information; The step of outputting the playback-related information to a network node different from the network node associated with the streaming client. Methods that include...

[0296] [EEE-B20] A device comprising a processor and a memory coupled to the processor and storing instructions for the processor, wherein the processor is adapted to perform the method described in any one of EEE-B1 to 19.

[0297] [EEE-B21] A program that, when executed by a processor, includes instructions that cause the processor to perform the method described in any one of EEE-B1 to 19.

[0298] [EEE-B22] A computer-readable storage medium that stores the program described in EEE-B21. [Prior art documents] [Patent Documents]

[0299] [Patent Document 1] Concurrently pending patent application: “Robust Intrusive Perceptual Audio Quality Assessment based on Convolutional Neural Networks,” applicant reference number D20118, U.S. provisional patent application 63 / 119,318, international application PCT / EP2021 / 083531, international publication WO / 2022 / 112594 [Non-patent literature]

[0300] [Non-Patent Document 1] Y. Leng, X. Tan, S. Zhao, F. Soong, X. -Y. Li and T. Qin, "MBNET: MOS Prediction for Synthesized Speech with Mean-Bias Network," ICASSP 2021, pp.391-395 [Non-Patent Document 2] W. -C. Huang, E. Cooper, J. Yamagishi, and T. Toda, "LDNet: Unified Listener Dependent Modeling in MOS Prediction for Synthetic Speech," ICASSP 2022, pp.896-900 [Non-Patent Document 3] G. Mittag, S. Zadtootaghaj, T. Michael, B. Naderi and S. Moeller, "Bias-Aware Loss for Training Image and Speech Quality Prediction Models from Multiple Datasets," 2021 13th International Conference on Quality of Multimedia Experience (QoMEX), pp.97-102 [Non-Patent Document 4] Zhi Li, Christos G. Bampis, Lucjan Janowski, Ioannis Katsavounidis, "A Simple Model for Subject Behavior in Subjective Experiments" in Proc. Int’l. Symp. on Electronic Imaging: Human Vision and Electronic Imaging, 2020, pp.131-1-131-14, https: / / doi.org / 10.2352 / ISSN.2470-1173.2020.11.HVEI-131

Claims

1. A method for evaluating playback performance in an adaptive streaming environment, the method being: The stage of obtaining playback-related information from the streaming client; A step of estimating the representation of a test audio signal based on the playback-related information, wherein the test audio signal is an audio signal played back by the streaming client; A step of determining an estimated value of the audio quality of the test audio signal based on an estimated representation of the test audio signal using an audio quality evaluation algorithm. Methods that include...

2. The step further includes generating a representation of a reference audio signal for the aforementioned test audio signal, The method according to claim 1.

3. The further step includes obtaining instructions for audio content processed by the streaming client from the streaming client, The method according to claim 1 or 2.

4. The method according to claim 3, wherein estimating the representation of the test audio signal is further based on the instructions of the audio content.

5. The method according to claim 3 or 4, as referenced to claim 2, wherein generating a representation of the reference audio signal is based on the instructions of the audio content.

6. The playback-related information includes bitrate information indicating the bitrate of the audio signal played back by the streaming client. Estimating the representation of the test audio signal is based on the bitrate information, The method according to any one of claims 1 to 5.

7. The aforementioned audio quality evaluation algorithm uses a set of pre-trained models for audio quality evaluation, Generating the estimated audio quality includes selecting a pre-trained model from the set of pre-trained models based on the playback-related information. The method according to any one of claims 1 to 6.

8. The aforementioned playback-related information includes information related to the playback device associated with the streaming client, The pre-trained model is selected based on information related to the playback device. The method according to claim 7.

9. The audio quality evaluation algorithm is implemented by a deep neural network (DNN) for estimating subjective listening score indications for the representation of a test audio signal as an estimate of the audio quality, wherein the listening score is a score from a predefined listening test, and the DNN is: An input stage for receiving a representation of the test audio signal and a representation of a reference audio signal for the test audio signal; Multiple layers for performing processing based on the representation of the test audio signal and the representation of the reference audio signal; Includes an output stage for generating the instructions for the listening score, The method according to any one of claims 1 to 8.

10. The aforementioned DNN, in one of several training epochs: The steps include: inputting one or more training data items that represent the respective values ​​of the aforementioned listening scores; A step of determining the indication for each of the listening scores based on one or more of the training data items; A step of determining the respective loss value for one or more training data items by evaluating the loss function, wherein the loss function depends on the indication of the listening score; By performing the step of adjusting one or more internal parameters of the DNN based on the determined loss value, The method according to claim 9, which is constructed by training the aforementioned DNN.

11. The method according to any one of claims 1 to 10, wherein the method is performed on a network node different from the streaming client.

12. The method according to any one of claims 1 to 11, wherein the representation of the estimated value of the test audio signal relates to one or more gammatone spectrograms.

13. The method according to claim 2 or any one of claims 3 to 12, as referenced from claim 2, wherein the representation of the reference audio signal relates to one or more gammatone spectrograms.

14. The method according to any one of claims 1 to 13, further comprising outputting an estimate of the audio quality of the test audio signal to a network node different from the network node associated with the streaming client.

15. The estimated value of the audio quality of the test audio signal is output to a network node to perform encoding and / or packaging of the audio content. The method further includes the step of optimizing the encoding and / or packaging based on the estimated value of the audio quality of the test audio signal. The method according to any one of claims 1 to 14.

16. The method according to claim 15, further comprising determining an optimal number of quality levels in a bitrate ladder for distribution by a content delivery network based on the estimated value of the audio quality of the test audio signal.

17. The method according to claim 15 or 16, further comprising determining the configuration and / or set of encoding tools based on the estimated value of the audio quality of the test audio signal.

18. The steps include: determining an estimate of the audio quality of the test audio signal for each streaming client in a group of multiple streaming clients; A step of comparing the estimated audio quality values ​​determined for the aforementioned group of streaming clients. The method according to any one of claims 1 to 17, further comprising:

19. A method for providing playback-related information in a streaming client that processes audio content in an adaptive streaming environment, the method being: Analyzing the playback buffer associated with the streaming client in order to determine bitrate information indicating the bitrate of the audio signal segment played back by the streaming client; Analyzing the manifest information associated with the aforementioned audio content; and By analyzing one or more of the characteristics of the playback device associated with the aforementioned streaming client The step of generating the aforementioned playback-related information; The step of outputting the playback-related information to a network node different from the network node associated with the streaming client. Methods that include...

20. An apparatus comprising a processor and a memory coupled to the processor and storing instructions for the processor, wherein the processor is adapted to perform the method according to any one of claims 1 to 19.

21. A program that, when executed by a processor, includes instructions causing the processor to perform the method according to any one of claims 1 to 19.

22. A computer-readable storage medium storing the program described in claim 21.

Images