Loss-conditional training and use of neural networks for audio processing using neural networks

Abstract

A computer implementation method for loss conditional training of a neural network for outputting an upsampled audio signal, the method comprising: randomly sampling a coefficient vector from a coefficient distribution, wherein elements of the coefficient vector represent weight coefficients corresponding to loss terms of a loss function; adjusting the neural network based on the coefficient vector; and training the adjusted neural network based on an audio training signal, wherein the training involves calculating the loss function of the audio training signal after processing by the adjusted neural network using the weight coefficients indicated by the coefficient vector.

Description

Technical Field

[0001] Cross - References to Related Applications This application claims the benefit of priority to U.S. Provisional Application No. 63 / 350,099, filed on June 08, 2022, and European Patent Application No. 22177849.1, filed on June 08, 2022, all of which are hereby incorporated by reference herein.

[0002] Techniques The present disclosure generally relates to methods for loss - conditional training of neural networks. In particular, coefficient vectors are randomly sampled from a distribution of coefficients, and the neural network is adjusted based on the coefficient vectors. The present disclosure further relates to computer - implemented methods for processing audio signals using loss - conditionally trained neural networks. The present disclosure further relates to respective devices and respective computer program products.

[0003] Some embodiments are described herein with particular reference to their disclosure, but it will be understood that the present disclosure is not limited to such fields of use and is applicable in a broader context.

Background Art

[0004] Any discussion of background art throughout this disclosure should not be construed as an admission that such art is widely known or forms part of common general knowledge in the field.

[0005] Audio quality as perceived by humans is a core performance metric in many audio devices. An audio codec is a computer program designed to encode and decode digital audio streams. More precisely, it compresses digital audio data into and decompresses it using codec algorithms. Audio codecs are intended to reduce storage space and bandwidth while maintaining high fidelity of the transmitted signal. However, lossy compression methods introduce encoding artifacts that can degrade audio quality.

[0006] Deep learning approaches are becoming increasingly attractive in a variety of application areas, including audio enhancement. Most deep learning approaches to date have focused on speech noise reduction.

[0007] Intuitively, regarding noise reduction in general, one might think that encoding artifact reduction and noise reduction are closely related. However, removing encoding artifacts / noise that are highly correlated with the desired sound often seems more complex than removing other types of noise (in noise reduction applications) that are less correlated. The characteristics of encoding artifacts depend on the codec, the encoding tool used, and the selected bitrate. In addition, modeling audio signals containing tone-like content such as speech and music is even more complex due to the periodic functions naturally present in these types of signals.

[0008] However, deep convolutional models used to reduce coding artifacts and coding noise are very complex in terms of model parameters and / or memory usage, and thus introduce a high computational load. Furthermore, when it is necessary to cover different signal categories such as speech, music, a mixture of speech and music, and applause, as well as various bitrates and codecs, typically separate models are trained, with each model providing the best possible performance for each task. [Overview of the project] [Problems that the invention aims to solve]

[0009] In light of the above, there is a need to improve the single model for more arbitrary inputs that cover different categories and conditions. [Means for solving the problem]

[0010] According to a first aspect of this disclosure, a method (e.g., a computer-implemented method) for loss-conditional training of a neural network to output an improved audio signal is provided. The method may include randomly sampling a coefficient vector from a distribution of coefficients. The elements of the coefficient vector may represent weight coefficients of a loss function. The weight coefficients may correspond to the loss term of the loss function. The method may further include conditioning the neural network based on the coefficient vector. The method may also include training the conditioned neural network based on an audio training signal, the training may involve calculating a loss function for the audio training signal after processing by the conditioned neural network using the weight coefficients represented by the coefficient vector.

[0011] In some embodiments, the loss function may be a multi-objective loss function.

[0012] In some embodiments, the distribution of the coefficients may be a uniform distribution within a predetermined range.

[0013] In some embodiments, tuning a neural network may include feature-wise linear modulation (FiLM).

[0014] In some embodiments, randomly sampling coefficient vectors, tuning a neural network, and training the tuned neural network may form at least part of an epoch, and the method may further include performing two or more epochs for each set of audio content types.

[0015] In some embodiments, training a tuned neural network can be performed in perceptually weighted regions.

[0016] In some embodiments, the neural network may implement a deep learning-based generator, which comprises an encoder stage and a decoder stage, each comprising multiple layers, each having one or more filters, and the last layer of the encoder stage is mapped to a latent feature space.

[0017] In some embodiments, the generator may be trained in a generative adversarial network (GAN) setting that includes a generator and a discriminator.

[0018] In some embodiments, tuning a neural network may involve tuning one or more layers of the encoder stage of a generator adjacent to the latent feature space.

[0019] In some embodiments, training a tuned neural network can be done as follows: Inputting an audio training signal to the aforementioned adjusted generator; The adjusted generator generates a processed audio training signal based on the audio training signal; The processed audio training signal and the corresponding original audio signal from which the audio training signal was derived are input to the discriminator one at a time; The discriminator determines whether the input audio signal is the processed audio training signal or the original audio signal; This may include sequentially and iteratively tuning the parameters of the generator until the discriminator can no longer distinguish the processed audio training signal from the original audio signal.

[0020] In some embodiments, a random noise vector z may be further applied to the latent feature space to modify the audio.

[0021] A second aspect of this disclosure provides a computer-implementable method for processing an audio signal using a loss-conditionally trained neural network. The method may include tuning the neural network based on tuning information, which includes a coefficient vector. The elements of the coefficient vector may represent weight coefficients of a loss function. The weight coefficients may correspond to the loss term of the loss function. The method may further include inputting the audio signal into the tuned neural network for processing the audio signal. The method may further include processing the audio signal by the tuned neural network based on the tuning information. The method may also include obtaining an improved audio signal as output from the tuned neural network.

[0022] In some embodiments, the loss function may be a multi-objective loss function.

[0023] In some embodiments, the adjustment information may be based on the content type and / or bitrate of the audio signal.

[0024] In some embodiments, adjusting the neural network may include Feature-wise Linear Modulation (FiLM).

[0025] In some embodiments, the neural network may implement a deep learning-based generator, which includes an encoder stage and a decoder stage, each including multiple layers with one or more filters in each layer, and the last layer of the encoder stage is mapped to a latent feature space.

[0026] In some embodiments, adjusting the neural network may involve adjusting one or more layers of the encoder stage of the generator adjacent to the latent feature space.

[0027] In some embodiments, a random noise vector z may be applied to the latent feature space to modify the audio.

[0028] In some embodiments, the method may further include receiving an audio bitstream including the audio signal and adjustment information.

[0029] In some embodiments, the method may further include core-decoding the audio bitstream to obtain the audio signal.

[0030] In some embodiments, the method may further include extracting adjustment information from the received bitstream.

[0031] In some embodiments, the method may further include analyzing the audio signal and determining adjustment information based on the result of the analysis.

[0032] In some embodiments, the method may be performed in a perceptually weighted region, and the improved audio signal in the perceptually weighted region may be obtained as the output from the tuned neural network.

[0033] In some embodiments, the method may further include converting the enhanced audio signal from a perceptually weighted region to the original signal region.

[0034] In some embodiments, the neural network may be trained in perceptually weighted regions.

[0035] According to a third aspect of this disclosure, an apparatus is provided for processing an audio signal using a loss-conditionally trained neural network. The apparatus may include one or more processors configured to perform a method, the method being: This involves tuning a neural network based on tuning information including a coefficient vector, wherein the elements of the coefficient vector may represent the weight coefficients of the loss function, and the weight coefficients may correspond to the loss term of the loss function; To process the aforementioned audio signal, the audio signal is input to a tuned neural network; A tuned neural network processes audio signals based on tuning information; This includes obtaining an improved audio signal as the output from a tuned neural network.

[0036] According to a fourth aspect of this disclosure, a computer program is provided, which, when executed by a computing device, includes instructions causing the computing device to perform the loss-conditional training method of a neural network described herein.

[0037] A fifth aspect of this disclosure provides a computer-readable storage medium for storing the computer program.

[0038] According to a sixth aspect of this disclosure, a computer program is provided which, when executed by a computing device, includes instructions causing the computing device to perform a method of processing an audio signal using a loss-conditionally trained neural network as described herein.

[0039] According to a seventh aspect of this disclosure, a computer-readable storage medium for storing the computer program is provided. [Brief explanation of the drawing]

[0040] Exemplary embodiments of the present disclosure are described below, merely as examples, with reference to the accompanying drawings. [Figure 1] This document presents an example of a loss-conditional training method for neural networks. [Figure 2] This example demonstrates loss-conditional training in a generative adversarial network (GAN) setting that includes generators and discriminators. [Figure 3] A schematic example of a simple generator architecture is shown below. [Figure 4] This example shows a computer implementation of processing audio signals using a loss-conditionally trained neural network. [Figure 5] Further examples of computer implementations that use loss-conditionally trained neural networks to process audio signals are presented. [Figure 6] An example of a device containing one or more processors is shown. [Modes for carrying out the invention]

[0041] Overview In deep learning-based techniques for improving (encoded) audio, the performance of a neural network (model) generally depends on several characteristics rather than a single one. The approach to training a neural network is to balance these characteristics by minimizing a loss function, which is a weighted sum of terms measuring various characteristics. Depending on the coefficients of these weights, training with this loss function yields a model best suited to a particular content type, bitrate, or codec.

[0042] However, when it is desired to cover different signal categories, such as speech, music, a mixture of speech and music, applause, and different bitrates and codecs, typically several separate neural networks with different weighting coefficients in the loss function are trained to achieve the best possible performance for each category. This requires maintaining multiple neural networks both during training and inference, and is computationally expensive in both cases.

[0043] The methods and apparatus described herein propose a loss-conditional training and inference strategy that allows for the training and inference of a single neural network for tasks that would normally require a large set of separately trained neural networks. This is based on the idea that if all of these separate neural networks are solving a highly related problem, some information can be shared among them. The loss function has coefficients to be tuned, and the described methods allow for the training and use of a single neural network covering a wide range of these coefficients. This provides a simple way to avoid inefficiencies and cover all trade-offs using a single neural network when a set of neural networks optimized for different losses would normally be required. That is, by changing the conditioning values, an improved output can be produced using a single neural network.

[0044] Loss-conditional training methods for neural networks Referring to the example in Figure 1, a method of loss-conditional training for a neural network is illustrated. Loss-conditional training may involve conditioning a neural network (model) to a loss function having weight coefficients λ corresponding to the loss terms. For example, the loss function may consist of four terms, three reconstruction terms, and an adversarial term responsible for the generative properties of the neural network. The reconstruction terms control how similar the improved signal produced by the neural network should be to the original signal, while the adversarial loss defines the amount of generative features that should be carried over to the improved signal. Thus, the weight coefficients λ are sometimes said to balance / control the ratio of terms in the loss function. That is, there may be a single set (ordered set) of weight coefficients λ that determines the optimized loss function for a single task / condition to be performed. Similarly, there may be a set of weight coefficients λ that determines an optimized loss function that works for multiple tasks / conditions. To help a neural network choose / discover the set of weight coefficients λ that best performs for multiple tasks / conditions, training may involve covering a wide range of different sets of weight coefficients. This can be achieved by sampling each vector from each distribution, as detailed below. Training may further involve covering different loss functions (determined by different weight coefficients) and / or different conditions, such as different content types / bitrates / codecs.

[0045] In step S101, a coefficient vector is randomly sampled from the coefficient distribution in order to train a single neural network to cover a wide range of coefficients. The elements of the coefficient vector may represent the weight coefficients of the loss function. It can be said that the coefficient vector represents an ordered set of the weight coefficients of the loss function. The number of weight coefficients in the set is determined by the number and / or weights of the terms in each loss function. Thus, it can be said that the distribution of coefficients represents the distribution of loss functions. This makes it possible to train a single neural network on a family of loss functions. The term loss function as used herein is also called a generator loss function.

[0046] In some embodiments, the generator loss function may be a multi-objective loss function, for example, the generator loss function may include a multi-resolution STFT loss function as given by equation (2) below. The results show that a multi-resolution STFT-based generator loss function provides quality improvements for processing diverse signal categories. In other words, if a single neural network is trained on diverse signal categories, which can be achieved, for example, by training the neural network on different audio training signals, quality improvements can be achieved if the coefficient vector is randomly sampled from the distribution of weight coefficients for the multi-objective loss function.

[0047] In some embodiments, the distribution of coefficients may be a uniform distribution within a predetermined range. That is, each element may be sampled from, for example, a (1D) distribution within the range [0, 100]. In this case, normalization can be said to be included as part of the weighting, so subsequent normalization of the vector is not required.

[0048] Referring again to the example in Figure 1, once the coefficient vector is sampled, in step S102, the neural network is tuned based on the coefficient vector. The tuning may be performed via a conditioning network; that is, the computations performed by the neural network may be tuned or modulated by the coefficient vector. In some embodiments, tuning the neural network may include feature-by-feature linear modulation (FiLM); that is, FiLM layers may be introduced into the architecture of the neural network, and these layers are parameterized by tuning based on the coefficient vector. For example, a randomly sampled conditioning vector λ may be fed into two multi-layer perceptron (MLP) networks, creating vectors σ(λ) and μ(λ) of the same dimension as the number of feature maps in the output of the convolutional / transpose layers being modulated / tuned. Each feature map is first scaled by σ(λ). Then, the scaled feature map is shifted by μ(λ).

[0049] Once the neural network is tuned, in step S103, the tuned neural network is then trained based on the audio training signal. Training may involve calculating a loss function for the audio training signal after processing by the tuned neural network, using weight coefficients indicated by coefficient vectors. In some embodiments, training the tuned neural network may be performed in a perceptually weighted region.

[0050] The method described above allows the neural network to learn to model an entire family of loss functions. The architecture and training of the neural network are described in more detail below.

[0051] Neural network architecture It should be noted that the method described above can be performed using any neural network, and therefore the architecture of the neural network is not limited. However, in some embodiments, the neural network may implement a deep learning-based generator, which includes an encoder stage and a decoder stage, each containing multiple layers with one or more filters in each layer, and the last layer of the encoder stage maps to a latent feature space.

[0052] A non-restrictive example of a simple generator architecture is schematically shown in the example in Figure 3. Generator 1000 includes an encoder stage 1001 and a decoder stage 1002. The encoder stage 1001 and decoder stage 1002 of generator 1000 may be fully convolutional. Decoder stage 1002 may mirror encoder stage 1001. Encoder stage 1001 and decoder stage 1002 may each include multiple layers 1001a, 1001b, 1001c, 1002a, 1002b, 1002c, each having multiple filters in each layer of the decoder stage, the multiple filters in each layer of the decoder stage can perform filtering operations that generate multiple feature maps, and the last layer of encoder stage 1001 can be mapped to a latent feature space representation c* 1003.

[0053] That is, the encoder stage 1001 and the decoder stage 1002 may each contain several L layers, each layer L having several N filters. L may be a natural number ≥ 1, and N may be a natural number ≥ 1. The size of the N filters (also known as the kernel size) is not limited, but the filter size may be the same in each of the L layers, for example, the filter size may be 31. The number of filters may increase in each layer. Each filter may operate on an audio signal input to each of the generator layers with a stride of 2, for example. Thus, learnable downsampling by factor 2 may be performed in the encoder layers, and learnable upsampling by factor 2 may be performed in the decoder layers. In other words, the encoder stage 1001 of the generator may contain multiple 1D convolutional layers (without bias terms) with a stride of 2, and the decoder stage 1002 of the generator may contain multiple 1D transposed convolutional layers (without bias terms) with a stride of 2.

[0054] In some embodiments, tuning a neural network may involve tuning one or more layers of the encoder stage of a generator adjacent to the latent feature space. In some embodiments, this tuning may be a FiLM tuning as described above.

[0055] In at least one layer of the encoder stage 1001, a nonlinear operation may be further performed as an activation including one or more of the following: a parametric rectified linear unit (PReLU), a rectified linear unit (ReLU), a leaky rectified linear unit (LReLU), an exponential linear unit (eLU), and a scaled exponential linear unit (SeLU). For example, the nonlinear operation may be based on a PReLU.

[0056] In one embodiment, the generator 1000 may further include a non-stride (stride=1) transposed convolutional layer as an output layer following the last layer 1002a of the decoder stage 1002. The output layer may include, for example, N=1 filters for a mono audio signal, and for example, N=2 filters for a stereo audio signal as an example of a multi-channel audio signal. The filter size may be 31. In the output layer, since the audio signal output from the decoder stage 1002 needs to be constrained to +1 and -1, the activation may be based on the tanh operation tanh(x) activation.

[0057] As shown in the example in Figure 3, in one embodiment, one or more skip connections 1005 may exist between the respective homogeneous layers of the encoder stage 1001 and decoder stage 1002 of the generator 1000. Here, the latent feature space representation c* 1003 is bypassed to prevent loss of information. The skip connections 1005 may be implemented using one or more of concatenation and signal addition. Due to the implementation of the skip connections 1005, the number of filter outputs may be "substantially" doubled.

[0058] In one embodiment, a random noise vector z(1004) may be further applied to a latent feature space representation c*(1003) to modify the audio.

[0059] Discriminator architecture As described above, in some embodiments, the generator may be a generator trained in a generative adversarial network setting (GAN setting). A GAN setting generally includes a generator G and a discriminator D, which are trained by a sequential iterative process. The discriminator architecture may have the same structure as the encoder stage of the generator. In other words, the discriminator architecture may mirror the structure of the encoder stage of the generator, but without modification. That is, the discriminator may include multiple layers, each having multiple filters. That is, the discriminator may include several L layers, with several N filters in each layer L. L may be a natural number ≥ 1, and N may be a natural number ≥ 1. The size of the N filters (also known as the kernel size) is not limited, but the filter size may be the same in each of the L layers, for example, the filter size may be 31. In each layer, the number of filters may increase. Each filter may operate, for example, on an audio signal input to each layer of the discriminator with a stride of 2. In other words, the discriminator may include multiple 1D convolutional layers with a stride of 2 (without a bias term). Nonlinear operations performed in at least one layer of the discriminator may include LReLU. Prepending on this, the discriminator may include an input layer. The input layer may be a non-stride convolutional layer (stride=1 means non-stride). The discriminator may further include an output layer. The output layer may have an N=1 filter with a filter size of 1 (the discriminator makes a single truth / false decision). Here, the filter size of the output layer may differ from the filter size of the discriminator layer. Thus, the output layer may be a one-dimensional (1D) convolutional layer that does not downsample hidden activations. This means that the filter in the output layer operates with a stride of 1, while all layers prior to the discriminator may use a stride of 2. The activation in the output layer may differ from the activation in at least one of the discriminator layers. The activation may be a sigmoid.However, when least-squares training methods are used, sigmoid activation is not required and is therefore optional.

[0060] Loss-conditional training method for generators in Generative Adversarial Network (GAN) settings The following describes loss-conditional training of a generator in a generative adversarial network (GAN) setting. Generally, during training in a generative adversarial network setting, the generator G generates a processed audio training signal x* by mapping it to a latent feature space representation using an encoder stage and upsampling the latent feature space representation using a decoder stage. The audio training signal can be derived from the original audio signal x, which is encoded and decoded, respectively. A random noise vector may be applied to the latent feature space representation. However, the random noise vector may be set to z=0. Setting the random noise vector to z=0 may yield the best results for reducing encoding artifacts. Alternatively, training may be performed without inputting the random noise vector z.

[0061] During training, the generator G attempts to output a processed audio training signal x* that is indistinguishable from the original audio signal x. The discriminator D is supplied with the generated processed audio training signal x* and the original audio signal x one at a time and determines whether the input signal is the processed audio training signal x* or the original audio signal x in a false / real manner. Here, the discriminator D attempts to discriminate the original audio signal x from the processed audio training signal x*. Throughout the sequential iteration process, the generator G tunes its parameters to produce an increasingly better processed audio training signal x* compared to the original audio signal x, and the discriminator D learns to better distinguish between the processed audio training signal x* and the original audio signal x. This adversarial learning process can be described by equation (1) below.

number

[0062] It should be noted that discriminator D may initially be trained to train generator G in the final step. Training and updating discriminator D may involve maximizing the probability of assigning a high score to the original audio signal x and a low score to the processed audio training signal x*. The goal in training discriminator D may be to recognize the original audio signal (unencoded) as genuine, while the processed audio training signal x* (generated) is recognized as fake. The parameters of generator G may be kept fixed while discriminator D is being trained and updated.

[0063] Next, training and updating the generator G may involve minimizing the difference between the original audio signal x and the generated processed audio training signal x*. The goal when training the generator G may be to achieve the discriminator D recognizing the generated processed audio training signal x* as genuine.

[0064] Referring to the example in Figure 2, loss-conditional training of a generator in a generative adversarial network (GAN) setting including a generator and a discriminator is shown. Loss-conditional training of generator G(100) may include the following:

[0065] The coefficient vector λ is randomly sampled from the coefficient distribution p(t). The distribution may be uniform in the range [0, 100]. The coefficient vector can be said to represent an ordered set of the weight coefficients λ of each loss function. The number of weight coefficients in the set is determined by the number and / or weighting of terms in each loss function. Thus, the coefficient distribution can be said to represent the distribution of the loss function. In one example, the elements of the coefficient vector may be independently sampled from a given range (e.g., the range [0, 100]).

[0066] Once the coefficient vector is sampled, the generator G 100 is then tuned based on the coefficient vector 108. The tuning may include feature-by-feature linear modulation (FiLM). In some embodiments, tuning the neural network may involve tuning one or more layers of the encoder stage of the generator adjacent to the latent feature space. That is, for example, only the one or more layers of the encoder stage may be modulated by FiLM tuning as described above.

[0067] An audio training signal (x with a tilde) 103 and optionally a random noise vector z 104 may be input to a (FiLM) tuned generator G 100. In one embodiment, the random noise vector z may be set to z=0. Alternatively, training may be performed without inputting the random noise vector z.

[0068] The audio training signal (x with a tilde) 103 can be obtained by encoding and decoding the original audio signal x 102. Then, based on the input, the tuned generator G 100 generates the processed audio training signal x* 105 by mapping it to a latent feature space representation using the encoder stage and upsampling the latent feature space representation using the decoder stage.

[0069] One at a time, the original audio signal x 102 from which the audio training signal (x with a tilde) 103 is derived, and the generated processed audio training signal x* 105 are input to discriminator D 101. As additional information, the audio training signal (x with a tilde) 103 may also be input to discriminator D 101 each time. Discriminator D 101 then determines whether the input data is the processed audio training signal x* 105 (false) or the original audio signal x 102 (true) (106).

[0070] In the next step, the parameters of the tuned generator G 100 are adjusted until the discriminator D 101 can no longer distinguish the processed audio training signal x* 105 from the original audio signal x 102. This may be done in a sequential iteration process 107.

[0071] The decision 101 made by discriminator D may be based on one or more perceptually motivated objective functions that follow the following equation (2).

number

[0072] As can be seen from the first term in equation (2), the adjusted generative adversarial network setting is applied by inputting the audio training signal (x with a tilde) as additional information to the discriminator.

[0073] The two terms in equation (2) above (of the generator loss function), which include coefficients λ2 and λ3, are sometimes called multi-resolution STFT loss terms. The multi-resolution STFT loss can be described as the sum of different STFT-based loss functions using different STFT parameters. sc m (spectral convergence loss) and L mag mThe (logarithmic scale STFT size loss) can be applied to STFT-based losses at M different resolutions, each with a number of FFT bins ∈{512, 1024, 2048}, a hop size ∈{50, 120, 240}, and finally a window length ∈{240, 600, 1200}. The results show that the multi-resolution STFT loss term provides quality improvements for general audio (i.e., arbitrary content type).

[0074] The term containing the coefficient λ1 in equation (2) is a 1-norm distance scaled by the factor lambda λ1. The value of this lambda may be selected from 1 to 100, depending on the application and / or the signal length input to the generator. For example, λ1 may be chosen such that λ1 = 100. Furthermore, the scaling for the multi-resolution STFT loss term may be set to the same value as the lambda.

[0075] term

number

[0076] Alternatively, the generator loss function in equation (2) may be applied in the following form:

number

[0077] In the above formula,

number

[0078] In some embodiments, training a tuned neural network (tuned generator) may be performed in a perceptually weighted region. A perceptually weighted audio training signal (x with a tilde) may then be input to the tuned generator G 100. The perceptually weighted audio training signal (x with a tilde) may be obtained by encoding and decoding the original perceptually weighted audio signal x. The original perceptually weighted audio signal x is derived by applying a mask or masking curve P to the original audio signal, the mask or masking curve representing a masking threshold derived from a psychoacoustic model. Based on the input, the tuned generator G 100 then generates a processed perceptually weighted audio training signal x*. In this case, the original perceptually weighted audio signal x from which the perceptually weighted audio training signal (x with a tilde) was derived, and the generated processed perceptually weighted audio training signal x* are input to the discriminator D 101.

[0079] In some embodiments, randomly sampling coefficient vectors, tuning the neural network, and training the tuned neural network may constitute at least part of an epoch, and the method may further include performing two or more epochs for each set of audio content types. The set of audio content types may include, for example, one or more of speech, music, speech and music, and applause. When performing two or more epochs for each set of audio content types, the neural network may be trained under multiple conditions. During training under multiple conditions, the neural network "looks" at the distribution of weight coefficients and "selects" the best coefficient vector that works best across the multiple conditions. Note that, in addition to audio content types, the multiple conditions may also include one or more of a set of bitrates and a set of codecs.

[0080] For example, in each epoch, the original audio signal is taken from a set of audio samples associated with the content type. Then, a coefficient vector is randomly sampled and used to tune the neural network (generator). The tuned neural network then processes each audio training signal, and a loss function is calculated for the processed audio training signal. Here, the weight coefficients of the loss function are given by the coefficient vector λ. Training then continues until the value of the calculated loss function is minimized as described above. That is, in each epoch, for each pair of the audio training signal derived from the original audio signal taken from the set of audio samples and the randomly sampled coefficient vector, the parameters of the tuned generator are tuned until the discriminator can no longer distinguish the processed audio training signal from the original audio signal. A single model can be trained to cover different signal categories when running two or more epochs for each set of audio content types. However, for each signal category, the training yields the best possible coefficient vector. Similarly, one or more epochs can be run for sets of bitrates and / or codecs. For example, a neural network may be initially trained on a set of audio content types, as described above. The same neural network may then be further trained on a set of bitrates and / or codecs, or vice versa. This training method yields a single coefficient vector that performs best for the multiple conditions experienced during training. However, this training method also allows a single neural network during inference to "remember" which coefficient vector yielded the best results during training for one particular condition (bitrate / codec / content type).During inference, this vector may then be selected and included in the conditioning information based on the respective bitrate / content type / codec.

[0081] Referring again to the example in Figure 2, generally, training of discriminator D 101 can follow the same general process as described above for training of generator G 100, but in this case the parameters of generator G 100 may be fixed, while the parameters of discriminator D 101 may be changed. Training of discriminator D 101 can be described by the following equation (3), which enables discriminator D 101 to discriminate the processed audio training signal x* 105 as false.

number

[0082] In the above case, the least squares approach (LS) and a modified generative adversarial network setting are also applied by inputting an audio training signal (x with a tilde) as additional information to discriminator D 101.

[0083] In addition to the least squares method, other training methods may be used to train tuned generators and discriminators in a generative adversarial network setting. This disclosure is not limited to any particular training method. Alternatively or additionally, the so-called Wasserstein approach may be used. In this case, instead of the least squares distance, the earthmover distance, also known as the Wasserstein distance, may be used. Generally, different training methods make the training of tuned generators and discriminators more stable. However, the type of training method applied does not affect the architecture of the (tuned) generator.

[0084] How to process audio signals using a loss-conditionally trained neural network Referring to the example in Figure 4, a computer implementation method for processing an audio signal using a loss-conditionally trained neural network is shown. In step S201, the neural network is tuned based on tuning information, which includes a coefficient vector. In some embodiments, as described above, the elements of the coefficient vector may represent the weight coefficients of the loss function. In some embodiments, the loss function may be a multi-objective loss function, such as the multi-resolution STFT-based generator loss function of equation (2), for example.

[0085] In some embodiments, the adjustment information may be based on (or determined by) the content type and / or bitrate of the audio signal to be processed. The adjustment information may further depend on the training results, for example, on which coefficient vector performed best. As described above, a loss-conditionally trained neural network may be trained by performing two or more epochs for each set of audio content types so that the neural network can cover a wide variety of different categories and conditions. Similarly, this may be applicable to the bitrate and / or codec; that is, the adjustment information may be based on the codec, either alternatively or additionally.

[0086] In some embodiments, tuning the neural network may include feature-by-feature linear modulation (FiLM), as described above. In some embodiments, the neural network may implement a deep learning-based generator. The generator may include an encoder stage and a decoder stage, each containing multiple layers with one or more filters in each layer, the last layer of the encoder stage mapping to a latent feature space. Tuning the neural network may then involve tuning one or more layers of the encoder stage of the generator adjacent to the latent feature space.

[0087] Furthermore, in some embodiments, a random noise vector z may be applied to the latent feature space to modify the audio.

[0088] Referring to the example in Figure 5, in some embodiments, the method may further include receiving an audio bitstream (200). The audio bitstream (200) includes an audio signal and tuning information. In this case, the method may further include, for example, core decoding the audio bitstream by an audio decoder 201 to obtain an audio signal 203. The method may also include extracting tuning information 204 from the received bitstream 200. In one exemplary embodiment, the audio bitstream may include metadata, the tuning information may be included in the metadata, or the metadata may indicate the tuning information. The trained neural network 202 may then be tuned based on the extracted tuning information 204. The audio signal 203 and optionally a random noise vector z 205 may then be input to the trained and tuned neural network 202 to process the audio signal 203. The processed audio signal 206 is obtained as output from the trained and tuned neural network 202.

[0089] Alternatively, in some embodiments, the method may further include analyzing an audio signal (203) and determining adjustment information based on the results of the analysis (204).

[0090] Referring again to the example in Figure 4, in step S202, the audio signal is input to the tuned neural network for processing the audio signal. In step S203, the tuned neural network processes the audio signal based on the tuning information. Then, in step S204, the processed audio signal is obtained as the output from the tuned neural network.

[0091] In some embodiments, the method may be performed in a perceptually weighted region. The processed audio signal in the perceptually weighted region may then be obtained as the output from a tuned neural network. In this case, the method may further include converting the processed audio signal from the perceptually weighted region to the original signal region. In some embodiments, the neural network may be trained in the perceptually weighted region.

[0092] Referring to the example in Figure 6, the method described above can be implemented by a device for processing audio signals using a loss-conditionally trained neural network. The device 300 may include one or more processors 301 configured to perform the method described above.

[0093] Alternatively or additionally, the methods described herein may also be implemented by a computer program that, when executed by a computing device, includes instructions causing the computing device to perform the methods described herein. The computer program may be provided on a computer-readable storage medium that stores the computer program.

[0094] The results demonstrated that a neural network trained as described herein is capable of producing outputs (processed audio signals) for different content types / bitrates / codecs that are as good as the outputs produced by individual neural networks trained for each specific task. That is, the methods and apparatus described herein allow for the training and inference of a single neural network for tasks that would normally require a large set of separately trained neural networks. This one-to-many approach provides a simple method for improving training and inference efficiency. Specifically, this technique reduces memory consumption on the decoder side because only one neural network needs to be memorized.

[0095] interpretation Unless otherwise specified, as will be apparent from the following description, discussions using terms such as “process,” “calculate,” “determine,” and “analyze” throughout this disclosure refer to actions and / or processes of a computer or computing system or similar electronic computing device that manipulate and / or transform data that is represented as a physical quantity, such as an electronic quantity, into other data that is similarly represented as a physical quantity.

[0096] Similarly, the term “processor” can refer to any device or part of a device that processes electronic data from, for example, registers and / or memory and converts that electronic data into other electronic data that can be stored, for example, in registers and / or memory. A “computer,” “computing machine,” or “computing platform” may include one or more processors.

[0097] The methods described herein are executable by one or more processors that accept computer-readable (also called machine-readable) code, which includes a set of instructions that, when executed by one or more of the processors, perform at least one of the methods described herein. This includes any processor capable of executing a set of instructions (sequential or otherwise) that specify an action to be taken. Thus, an example is a typical processing system comprising one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem, which includes main RAM and / or static RAM, and / or ROM. A bus subsystem may be included for communication between components. The processing system may further be a distributed processing system having processors connected by a network. If the processing system requires a display, such a display may include, for example, a liquid crystal display (LCD) or a cathode ray tube (CRT) display. If manual data entry is required, the processing system also includes input devices, such as one or more of an alphanumeric input unit, such as a keyboard, and a pointing control device, such as a mouse. The processing system may also include a storage system, such as a disk drive unit. The processing system in some configurations may include an audio output device and a network interface device. Thus, the memory subsystem includes a computer-readable carrier medium carrying computer-readable code (e.g., software) containing a set of instructions for performing one or more of the methods described herein when executed by one or more processors. Note that, when a method includes several elements, e.g., several steps, the order of such elements is not implied unless otherwise specified. The software may reside on a hard disk, or, during its execution by the computer system, may reside entirely or at least partially in RAM and / or within the processor.Therefore, memory and processors also constitute computer-readable carrier media that carry computer-readable code. Furthermore, computer-readable carrier media may form or be included in computer program products.

[0098] In alternative exemplary embodiments, one or more processors may operate as standalone devices or be connected to other processors in a networked deployment, for example, by a network connection, and one or more processors may operate as a server or user machine in a server-user network environment, or as a peer machine in a peer-to-peer or distributed network environment. One or more processors may form any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be performed by a personal computer (PC), tablet PC, personal digital assistant (PDA), mobile phone, web appliance, network router, switch or bridge, or any other machine.

[0099] It should be noted that the term “machine” can also be interpreted as including any collection of machines that individually or jointly execute a set (or set) of instructions to perform any one or more of the methods described herein.

[0100] Therefore, each exemplary embodiment of the methods described herein may take the form of a computer-readable carrier medium carrying a set of instructions for execution on one or more processors, for example, one or more processors that are part of a web server configuration, such as a computer program. Thus, as will be understood by those skilled in the art, exemplary embodiments of the Disclosure may be embodied as a method, an apparatus such as a dedicated device, an apparatus such as a data processing system, or a computer-readable carrier medium, for example, a computer program product. The computer-readable carrier medium carries computer-readable code that, when executed on one or more processors, causes one or more processors to perform the method. Thus, aspects of the Disclosure may take the form of a method, an exemplary embodiment entirely of hardware, an exemplary embodiment entirely of software, or an exemplary embodiment combining software and hardware aspects. Furthermore, the Disclosure may take the form of a carrier medium (for example, a computer program product on a computer-readable storage medium) carrying computer-readable program code embodied on the medium.

[0101] The software may be further transmitted or received over a network via a network interface device. While the carrier medium is a single medium in exemplary embodiments, the term “carrier medium” should be interpreted to include a single or multiple mediums (e.g., a centralized or distributed database and / or associated caches and servers) that store one or more instruction sets. The term “carrier medium” should also be interpreted to include any medium capable of storing, encoding, or carrying a set of instructions for execution by one or more processors, causing one or more processors to execute any one or more of the methods of this disclosure. The carrier medium can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical disks, magnetic disks, and magneto-optical disks. Volatile media include dynamic memory, such as main memory. Transmission media include coaxial cables, copper wires, and optical fibers, and include wires, such as those used in bus subsystems. Transmission media can also take the form of sound waves or light waves, such as those generated during radio and infrared data communications. Therefore, the term “carrier medium” shall be interpreted as including, but not limited to, computer products embodied in solid memory, optical media and magnetic media; media carrying a propagating signal detectable by at least one processor or one or more processors and representing a set of instructions that, when executed, perform a method; and transmission media in a network carrying a propagating signal detectable by at least one of the one or more processors and representing a set of instructions.

[0102] It will be understood that, in one exemplary embodiment, the steps of the method discussed are performed by a suitable processor (or more processors) of a processing (e.g., a computer) system that executes instructions (computer-readable code) stored in a memory device. It will also be understood that this disclosure is not limited to any particular implementation or programming technique, and may be implemented using any suitable technique for implementing the functionality described herein. This disclosure is not limited to any particular programming language or operating system.

[0103] Throughout this disclosure, any reference to “certain embodiments,” “some embodiments,” or “exemplary embodiments” means that any particular feature, structure, or characteristic described in relation to that embodiment is included in at least one embodiment of this disclosure. Thus, the phrases “in certain embodiments,” “some embodiments,” or “exemplary embodiments” appearing in various places throughout this disclosure do not necessarily all refer to the same exemplary embodiment. Furthermore, any particular feature, structure, or characteristic may be combined in any suitable manner in one or more exemplary embodiments, as will be apparent to those skilled in the art from this disclosure.

[0104] Where used herein, unless otherwise specified, the use of ordinal adjectives such as “first,” “second,” “third,” etc., to describe a common subject simply indicates that different examples of similar subjects are being referred to, and is not intended to imply that the subjects described in this way must be in a given order, temporally, spatially, in ranking, or in any other way.

[0105] In the following claims and description herein, any term consisting of, having, or possessing is an open term meaning that includes at least the elements / features that follow it, but does not exclude others. Therefore, when used in the claims, the term including should not be interpreted as being limited to the enumerated means, elements, or steps. For example, the expression "device including A and B" should not be limited to a device consisting only of elements A and B. Any term including, containing, or encompassing as used herein is also an open term meaning that includes at least the elements / features that follow it, but does not exclude others. Therefore, including is synonymous with having and means having.

[0106] In the above description of exemplary embodiments of the Disclosure, please understand that various features of the Disclosure may be grouped together in a single exemplary embodiment, figure, or description thereof for the purpose of improving the flow of the Disclosure and aiding in the understanding of one or more of the various aspects of the invention. However, this method of disclosure should not be construed as reflecting an intention that the claims require more features than expressly described in each claim. Rather, as reflected in the following claims, the aspects of the invention are fewer than all the features of a single above-disclosed exemplary embodiment. Thus, the claims following this specification are expressly incorporated herein, and each claim stands alone as a separate exemplary embodiment of the Disclosure.

[0107] Furthermore, some exemplary embodiments described herein include some features included in other exemplary embodiments, but do not include other features, and combinations of features of different exemplary embodiments are intended to be within the scope of this disclosure and form different exemplary embodiments. This will be understood by those skilled in the art. For example, in the following claims, any of the claimed exemplary embodiments may be used in any combination.

[0108] Numerous specific details are provided in the description herein. However, it should be understood that exemplary embodiments of this disclosure may be carried out without these specific details. On the other hand, well-known methods, structures, and techniques are not described in detail in order to avoid ambiguity in understanding this paper.

[0109] Therefore, while what is considered to be the best form of this disclosure has been described, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of this disclosure. It is intended that all such changes and modifications that fall within the scope of this disclosure be requested. For example, each of the formulas given above is merely a representative example of a procedure that may be used. Functions may be added to or removed from the block diagram, and operations may be exchanged between function blocks. Steps may be added to or removed from the described method within the scope of this disclosure.

[0110] Various aspects and implementations of this disclosure can also be understood from the following enumerated example embodiments (EEEs) which are not part of the claims.

[0111] [EEE1] A method for loss-conditional training of a neural network, wherein the method is: The step involves randomly sampling a coefficient vector from the distribution of coefficients, wherein the elements of the coefficient vector represent the weight coefficients of the loss function; A step of adjusting the neural network based on the coefficient vector; A step of training a tuned neural network based on an audio training signal, wherein the training includes calculating the loss function for the audio training signal after processing by the tuned neural network using the weight coefficients indicated by the coefficient vector. Methods that include... [EEE2] The loss function is a multi-objective loss function, as described in EEE1. [EEE3] The method according to EEE1 or 2, wherein the distribution of the coefficients is a uniform distribution within a predetermined range. [EEE4] Tuning the neural network is the method described in any one of EEE1 to 3, including feature-wise linear modulation (FiLM). [EEE5] The method according to any one of EEE1 to 4, wherein randomly sampling the coefficient vector, tuning the neural network, and training the tuned neural network constitute at least part of an epoch, and the method further comprises performing two or more epochs for each set of audio content types. [EEE6] Training the aforementioned tuned neural network is performed in perceptually weighted regions, as described in any one of EEE1 to 5. [EEE7] The method according to any one of EEE1 to 6, wherein the neural network implements a deep learning-based generator, the generator comprising an encoder stage and a decoder stage, each comprising a plurality of layers having one or more filters in each layer, and the last layer of the encoder stage is mapped to a latent feature space. [EEE8] The method according to EEE6, wherein the generator is trained in a generative adversarial network (GAN) setting including the generator and the discriminator. [EEE9] The method according to EEE6, wherein tuning the neural network includes tuning one or more layers of the encoder stage of the generator adjacent to the latent feature space. [EEE10] Training the aforementioned modified neural network involves: Inputting an audio training signal to the aforementioned adjusted generator; The adjusted generator generates a processed audio training signal based on the audio training signal; The processed audio training signal and the corresponding original audio signal from which the audio training signal was derived are input to the discriminator one at a time; The discriminator determines whether the input audio signal is the processed audio training signal or the original audio signal; This includes sequentially and iteratively tuning the parameters of the generator until the discriminator can no longer distinguish the processed audio training signal from the original audio signal, The method described in EEE8 or 9. [EEE11] The method according to EEE10, wherein a random noise vector z is applied to the latent feature space to modify the audio. [EEE12] A computer-implemented method for processing an audio signal using a loss-conditionally trained neural network, the method being: A step of adjusting the neural network based on adjustment information including coefficient vectors; To process the aforementioned audio signal, the audio signal is input to a modified neural network; The steps include: processing the audio signal based on the adjustment information using the adjusted neural network; The step includes obtaining a processed audio signal as the output from the aforementioned adjusted neural network, method. [EEE13] The coefficient vector represents the weight coefficients of the loss function, as described in EEE12. [EEE14] The loss function is a multi-objective loss function, as described in EEE13. [EEE15] The adjustment information is the method described in any one of EEE12 to 14, based on the content type and / or bitrate of the audio signal. [EEE16] Tuning the neural network is the method described in any one of EEE12 to 15, including feature-by-feature linear modulation (FiLM). [EEE17] The method according to any one of EEE12 to 16, wherein the neural network implements a deep learning-based generator, the generator comprising an encoder stage and a decoder stage, each comprising a plurality of layers having one or more filters in each layer, and the last layer of the encoder stage is mapped to a latent feature space. [EEE18] The method according to EEE17, wherein tuning the neural network includes tuning one or more layers of the encoder stage of the generator adjacent to the latent feature space. [EEE19] The method according to EEE18 or 19, wherein a random noise vector z is applied to the latent feature space to correct the audio. [EEE20] The method according to any one of EEE12 to 19, further comprising receiving an audio bitstream containing the audio signal and the adjustment information. [EEE21] The method according to EEE20, further comprising core-decoding the audio bitstream to obtain the audio signal. [EEE22] The method according to EEE20 or 21, further comprising extracting the adjustment information from the received bitstream. [EEE23] The method according to any one of EEE12 to 21, further comprising analyzing the audio signal and determining the adjustment information based on the results of the analysis. [EEE24] The method according to any one of EEE12 to 23, wherein the method is performed in a perceptually weighted region, and the processed audio signal in the perceptually weighted region is obtained as the output from the tuned neural network. [EEE25] The method according to EEE24, further comprising converting the processed audio signal from the perceptually weighted region to the original signal region. [EEE26] The method according to any one of EEE12 to 25, wherein the neural network is trained in the perceptually weighted regions. [EEE27] An apparatus for processing an audio signal using a loss-conditionally trained neural network, the apparatus comprising one or more processors configured to perform a method, the method being: A step of adjusting the neural network based on adjustment information including coefficient vectors; To process the aforementioned audio signal, the steps include: inputting the audio signal into the adjusted neural network; The steps include: processing the audio signal based on the adjustment information using the adjusted neural network; The step includes obtaining a processed audio signal as the output from the aforementioned adjusted neural network, Device. [EEE28] A computer program, when executed by a computing device, that includes instructions causing the computing device to perform the method described in any one of EEE1 to 11. [EEE29] A computer-readable storage medium that stores computer programs as described in EEE28. [EEE30] A computer program, when executed by a computing device, that includes instructions causing the computing device to perform the method described in any one of the EEE12 to 26. [EEE31] A computer-readable storage medium that stores computer programs as defined in EEE30.

Claims

1. A loss-conditional training method for a neural network to output an improved audio signal, which is implemented on a computer, and the method is: The step of randomly sampling a coefficient vector from the distribution of coefficients, wherein the elements of the coefficient vector represent weight coefficients corresponding to the loss term of the loss function; A step of adjusting the neural network based on the coefficient vector; A step of training a tuned neural network based on an audio training signal, wherein the training includes calculating the loss function for the audio training signal after processing by the tuned neural network using the weight coefficients indicated by the coefficient vector. Methods that include...

2. The method according to claim 1, wherein the loss function is a multi-objective loss function.

3. The method according to claim 1, wherein the distribution of the coefficients is a uniform distribution within a predetermined range.

4. The method according to claim 1, wherein tuning the neural network includes feature-wise linear modulation (FiLM).

5. The method according to claim 1, wherein randomly sampling the coefficient vector, tuning the neural network, and training the tuned neural network constitute at least part of an epoch, and the method further comprises performing two or more epochs for each set of audio content types.

6. The method according to claim 1, wherein the training of the adjusted neural network is performed in perceptually weighted regions.

7. The method according to claim 1, wherein the neural network implements a deep learning-based generator, the generator comprising an encoder stage and a decoder stage, each comprising a plurality of layers having one or more filters in each layer, and the last layer of the encoder stage is mapped to a latent feature space.

8. The method according to claim 7, wherein tuning the neural network includes tuning one or more layers of the encoder stage of the generator adjacent to the latent feature space.

9. The method according to claim 7, wherein the generator is trained in a generative adversarial network (GAN) setting including the generator and the discriminator.

10. Training the aforementioned tuned neural network involves: Inputting an audio training signal to the aforementioned adjusted generator; The adjusted generator generates a processed audio training signal based on the audio training signal; The processed audio training signal and the corresponding original audio signal from which the audio training signal was derived are input to the discriminator one at a time; The discriminator determines whether the audio signal input to it is the processed audio training signal or the original audio signal; This includes sequentially and iteratively tuning the parameters of the generator until the discriminator can no longer distinguish the processed audio training signal from the original audio signal, The method according to claim 9.

11. The method according to claim 10, wherein a random noise vector z is applied to the latent feature space to modify the audio.

12. A computer-implemented method for processing an audio signal using a loss-conditionally trained neural network, the method being: A step of adjusting the neural network based on adjustment information including a coefficient vector, wherein the elements of the coefficient vector represent weight coefficients corresponding to the loss term of the loss function; To process the aforementioned audio signal, the audio signal is input to a modified neural network; The steps include: processing the audio signal based on the adjustment information using the adjusted neural network; The step includes obtaining an improved audio signal as the output from the adjusted neural network, method.

13. The method according to claim 12, wherein the loss function is a multi-objective loss function.

14. The method according to claim 12, wherein the adjustment information is based on the content type and / or bitrate of the audio signal.

15. The method according to claim 12, wherein tuning the neural network includes feature-by-feature linear modulation (FiLM).

16. The method according to claim 12, wherein the neural network implements a deep learning-based generator, the generator comprising an encoder stage and a decoder stage, each comprising a plurality of layers having one or more filters in each layer, and the last layer of the encoder stage is mapped to a latent feature space.

17. The method according to claim 16, wherein tuning the neural network includes tuning one or more layers of the encoder stage of the generator adjacent to the latent feature space.

18. The method according to claim 16, wherein a random noise vector z is applied to the latent feature space to modify the audio.

19. The method according to claim 12, further comprising receiving an audio bitstream containing the audio signal and the adjustment information.

20. The method according to claim 19, further comprising core-decoding the audio bitstream to obtain the audio signal.

21. The method according to claim 19, further comprising extracting the adjustment information from the received bitstream.

22. The method according to claim 12, further comprising analyzing the audio signal and determining the adjustment information based on the results of the analysis.

23. The method according to claim 12, wherein the method is performed in a perceptually weighted region, and the improved audio signal in the perceptually weighted region is obtained as the output from the tuned neural network.

24. The method according to claim 23, further comprising converting the improved audio signal from the perceptually weighted region to the original signal region.

25. The method according to claim 12, wherein the neural network is trained in perceptually weighted regions.

26. An apparatus for processing an audio signal using a loss-conditionally trained neural network, the apparatus comprising one or more processors configured to perform a method, the method being: A step of adjusting the neural network based on adjustment information including a coefficient vector, wherein the elements of the coefficient vector represent weight coefficients corresponding to the loss term of the loss function; The steps include: inputting the audio signal into the adjusted neural network in order to process the aforementioned audio signal; The steps include: processing the audio signal based on the adjustment information using the adjusted neural network; The step includes obtaining an improved audio signal as the output from the adjusted neural network, Device.

27. A computer program, when executed by a computing device, includes instructions that cause the computing device to perform the method according to any one of claims 1 to 11.

28. A computer-readable storage medium storing the computer program described in claim 27.

29. A computer program, when executed by a computing device, includes instructions that cause the computing device to perform the method according to any one of claims 12 to 25.

30. A computer-readable storage medium storing the computer program described in claim 29.

Images