Loss condition training and use of neural network to process audio using neural network

Abstract

A computer-implemented method of loss condition training of a neural network for outputting an enhanced audio signal, the method comprising: randomly sampling a coefficient vector from a distribution of coefficients, where elements of the coefficient vector indicate weight coefficients corresponding to loss terms of a loss function; carrying out conditional processing on the neural network based on the coefficient vector; and training the conditioned neural network based on the audio training signal, where the training involves calculating a loss function for the audio training signal processed by the conditioned neural network using the weight coefficients indicated by the coefficient vectors.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 350,099, filed on June 8, 2022, and European Application No. 22177849.1, filed on June 8, 2022, all of which are incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to methods for loss-conditioned training of neural networks. Specifically, coefficient vectors are randomly sampled from a distribution of coefficients, and the neural network is conditioned based on the coefficient vectors. The present disclosure also relates to a computer-implemented method for processing an audio signal using a loss-conditioned trained neural network. In addition, the present disclosure also relates to corresponding apparatus and corresponding computer program products.

[0004] Although some embodiments will be described herein with particular reference to this disclosure, it will be appreciated that the disclosure is not limited to such fields of use and is applicable in a much broader context. Background Art

[0005] Any discussion of the background art throughout the present disclosure should in no way be considered as an admission that this art is widely known or forms part of the common general knowledge in the field.

[0006] Audio quality as perceived by humans is a core performance metric for many audio devices. An audio codec is a computer program designed to encode and decode digital audio streams. More precisely, an audio codec compresses and decompresses digital audio data into and from a compressed format with the help of a codec algorithm. Audio codecs are designed to reduce storage space and bandwidth while maintaining high fidelity of the transmitted signal. However, lossy compression methods introduce coding artifacts that can impair audio quality.

[0007] Deep learning methods are becoming increasingly attractive in various application areas including audio enhancement. So far, most deep learning methods involve speech denoising.

[0008] For general denoising, one might intuitively assume that coding artifact reduction and denoising are highly correlated. However, removing coding artifacts / noise that is highly correlated with the desired sound appears to be more complex than removing other noise types (in noise reduction applications) that are generally less correlated. The characteristics of coding artifacts depend on the codec and the coding tools used, as well as the chosen bitrate. Furthermore, since periodic functions are naturally included in audio signals that include tonal content (such as speech and music), modeling such audio signals is even more complex.

[0009] However, deep convolutional models used to reduce coding artifacts and coding noise are rather complex in terms of model parameters and / or memory usage, thus introducing a high computational load themselves. Furthermore, if different signal categories, such as, for example, speech, music, a mixture of speech and music, and applause, as well as different bitrates and codecs need to be covered, separate models are usually trained, each providing the best possible performance for each task.

[0010] Therefore, in view of the above, there is an existing need to improve a single model to achieve more arbitrary inputs covering different categories and conditions. Summary of the invention

[0011] According to a first aspect of the present disclosure, a method (e.g., a computer-implemented method) for loss-conditioning training of a neural network that outputs an enhanced audio signal is provided. The method may include randomly sampling a coefficient vector from a distribution of coefficients, wherein the elements of the coefficient vector may indicate weight coefficients of a loss function. The weight coefficients may correspond to loss terms of the loss function. The method may also include conditioning the neural network based on the coefficient vector. And the method may include training the conditioned neural network based on an audio training signal, wherein the training may involve calculating the loss function of the audio training signal processed by the conditioned neural network using the weight coefficients indicated by the coefficient vector.

[0012] In some embodiments, the loss function may be a multi-objective loss function.

[0013] In some embodiments, the distribution of coefficients may be a uniform distribution within a predetermined range.

[0014] In some embodiments, conditioning the neural network may include Feature-wise Linear Modulation (FiLM).

[0015] In some embodiments, randomly sampling coefficient vectors, conditioning the neural network, and training the conditioned neural network may form at least a portion of an epoch, and the method may further include performing two or more epochs for each audio content type in a set of audio content types.

[0016] In some embodiments, training the conditioned neural network may be performed in the perceptually weighted domain.

[0017] In some embodiments, the neural network may implement a deep learning based generator comprising an encoder stage and a decoder stage, each stage comprising multiple layers with one or more filters in each layer, and the last layer of the encoder stage maps to a latent feature space.

[0018] In some embodiments, the generator can be trained in a generative adversarial network (GAN) setting including a generator and a discriminator.

[0019] In some embodiments, conditioning the neural network may involve conditioning on one or more layers of an encoder stage of the generator adjacent to the latent feature space.

[0020] In some embodiments, training a conditioned neural network may include:

[0021] Input the audio training signal into the conditioned generator;

[0022] generating, by the conditioned generator, a processed audio training signal based on the audio training signal;

[0023] Inputting the processed audio training signal and the corresponding original audio signal from which the audio training signal is obtained into the discriminator one by one;

[0024] determining, by a discriminator, whether the input audio signal is a processed audio training signal or an original audio signal; and

[0025] The parameters of the generator are iteratively adjusted until the discriminator can no longer distinguish between the processed audio training signal and the original audio signal.

[0026] In some embodiments, a random noise vector z may also be applied to the latent feature space to modify the audio.

[0027] According to a second aspect of the present disclosure, there is provided a computer-implemented method for processing an audio signal using a neural network trained with loss conditioning. The method may include conditioning the neural network based on conditioning information including a coefficient vector. The elements of the coefficient vector may indicate weight coefficients of a loss function. The weight coefficients may correspond to loss terms of the loss function. The method may also include inputting an audio signal into the conditioned neural network to process the audio signal. The method may also include processing the audio signal based on the conditioning information by the conditioned neural network. And, the method may include obtaining an enhanced audio signal as an output of the conditioned neural network.

[0028] In some embodiments, the loss function may be a multi-objective loss function.

[0029] In some embodiments, the conditioning information may be based on the content type and / or bit rate of the audio signal.

[0030] In some embodiments, conditioning the neural network may include feature linear modulation FiLM.

[0031] In some embodiments, the neural network may implement a deep learning based generator comprising an encoder stage and a decoder stage, each stage comprising multiple layers with one or more filters in each layer, and the last layer of the encoder stage maps to a latent feature space.

[0032] In some embodiments, conditioning the neural network may involve conditioning on one or more layers of an encoder stage of the generator adjacent to the latent feature space.

[0033] In some embodiments, a random noise vector z may be applied to the latent feature space to modify the audio.

[0034] In some embodiments, the method may further include receiving an audio bitstream including the audio signal and the conditioning information.

[0035] In some embodiments, the method may further include the core decoding the audio bitstream to obtain an audio signal.

[0036] In some embodiments, the method may further include extracting conditioning information from the received bitstream.

[0037] In some embodiments, the method may further include analyzing the audio signal and determining the conditioning information based on a result of the analysis.

[0038] In some embodiments, the method may be performed in a perceptually weighted domain, and an enhanced audio signal may be obtained in the perceptually weighted domain as an output of the conditioned neural network.

[0039] In some embodiments, the method may further include converting the enhanced audio signal from the perceptually weighted domain to the original signal domain.

[0040] In some embodiments, the neural network may have been trained in the perceptually weighted domain.

[0041] According to a third aspect of the present disclosure, there is provided an apparatus for processing an audio signal using a neural network trained using a loss condition. The apparatus may include one or more processors configured to perform a method including the following operations:

[0042] Conditioning the neural network based on conditioning information including a coefficient vector, wherein elements of the coefficient vector may indicate weight coefficients of a loss function, and the weight coefficients may correspond to loss terms of the loss function;

[0043] Inputting the audio signal into a conditioned neural network to process the audio signal;

[0044] processing the audio signal based on the conditioning information by the conditioned neural network; and

[0045] The enhanced audio signal is obtained as the output of the conditioned neural network.

[0046] According to a fourth aspect of the present disclosure, a computer program is provided, the computer program comprising instructions, which, when executed by a computing device, causes the computing device to perform the method for loss conditional training of a neural network described herein.

[0047] According to a fifth aspect of the present disclosure, a computer-readable storage medium storing the above-mentioned computer program is provided.

[0048] According to a sixth aspect of the present disclosure, a computer program is provided, which includes instructions. When the instructions are executed by a computing device, the computing device executes the method of processing an audio signal using a neural network trained with loss conditions as described herein.

[0049] According to a seventh aspect of the present disclosure, a computer-readable storage medium storing the above-mentioned computer program is provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Exemplary embodiments of the present disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0051] Figure 1 An example of a method for loss-conditional training of a neural network is illustrated.

[0052] Figure 2 Illustrated is an example of loss-conditioned training in a Generative Adversarial Network (GAN) setting consisting of a generator and a discriminator.

[0053] Figure 3 An example of a simple architecture of a generator is schematically illustrated.

[0054] Figure 4 An example of a computer-implemented method for processing an audio signal using a neural network trained with a loss condition is illustrated.

[0055] Figure 5 A further example of a computer-implemented method of processing an audio signal using a neural network trained with a loss condition is illustrated.

[0056] Figure 6 An example of an apparatus including one or more processors is illustrated. DETAILED DESCRIPTION

[0057] Overview

[0058] In deep learning based methods for (coded) audio enhancement, the performance of the neural network (model) usually does not depend on a single attribute, but on several. The way to train a neural network is to balance the different attributes by minimizing a loss function, which is a weighted sum of terms measuring these attributes. Depending on the coefficients of these weights, training with this loss function results in a model that is best suited for certain content types, bitrates or codecs.

[0059] However, if one wishes to cover different signal categories, such as speech, music, a mixture of speech and music, and applause, as well as different bitrates and codecs, one typically needs to train several separate neural networks with different weight coefficients in the loss function to obtain the best possible performance for each category. This requires keeping multiple neural networks in parallel during training and inference, which is computationally expensive at both training and inference time.

[0060] The methods and apparatus described herein propose a loss-conditioned training and inference strategy that allows a single neural network to be trained and inferred for tasks that would normally require a large set of separately trained neural networks. This is based on the idea that if all of these individual neural networks solve very related problems, then some information can be shared between them. The loss function has coefficients that need to be adjusted, and the described method allows a single neural network to be trained and used that covers a wide range of these coefficients. This provides a simple way to avoid inefficiencies and use a single neural network to cover all trade-offs in situations where a set of neural networks would normally be required to optimize for different losses. That is, by changing the conditioning values, an enhanced output can be generated with a single neural network.

[0061] Method for loss-conditional training of neural networks

[0062] refer to Figure 1, illustrating a method for loss-conditioned training of a neural network. Loss-conditioned training may involve conditioning a neural network on a loss function having a weight coefficient λ corresponding to a loss term. For example, the loss function may consist of four terms responsible for the generative properties of the neural network, namely three reconstruction terms and one adversarial term. The reconstruction term regulates how similar the enhanced signal generated by the neural network should be to the original signal, while the adversarial loss defines the amount of generated features that should be carried into the enhanced signal. Therefore, the weight coefficient λ can be said to balance / regulate the proportion of each term in the loss function. That is, for a single task / single condition to be performed, there may be a single (ordered) set of weight coefficients λ that determine the optimized loss function for that task / condition. In a similar manner, there may also be a set of weight coefficients λ that determine the optimized loss function applicable to multiple tasks / conditions. In order for the neural network to select / find a set of weight coefficients λ that best fit multiple tasks / conditions, training may involve covering a wide range of different sets of weight coefficients, which may be achieved by sampling corresponding vectors from corresponding distributions, as described below. Training may also involve covering different loss functions (determined by different weight coefficients) and / or different conditions, such as different content types / bitrates / codecs.

[0063] In order to train a single neural network to cover a wide range of coefficients, in step S101, a coefficient vector is randomly sampled from the distribution of coefficients. The elements of the coefficient vector may indicate weight coefficients of a loss function. It can be said that the coefficient vector represents an ordered group of weight coefficients of a loss function. The number of weight coefficients in a group is determined by the number of terms and / or weights in the corresponding loss function. Therefore, it can be said that the distribution of coefficients represents the distribution of loss functions. This enables training a single neural network on a family of loss functions. The term "loss function" used here can also be referred to as a generator loss function.

[0064] In some embodiments, the generator loss function may be a multi-objective loss function, for example, the generator loss function may include a multi-resolution STFT loss function as given by formula (2) below. It is shown that the generator loss function based on the multi-resolution STFT provides quality improvements for processing various signal categories. In other words, if, for example, a single neural network is trained on various signal categories, which can be achieved by training the neural network on different audio training signals, then quality improvements can be achieved if the coefficient vector is randomly sampled from the distribution of weight coefficients of the multi-objective loss function.

[0065] In some embodiments, the distribution of coefficients may be a uniform distribution within a predetermined range. That is, each element may be sampled, for example, from a (1D) distribution in the range [0, 100]. In this case, subsequent normalization of the vector is not required, since the normalization can be said to be included as part of the weight.

[0066] Reference again Figure 1 In the example of , once the coefficient vector is sampled, in step S102, the neural network is conditioned based on the coefficient vector. The conditioning can be performed via a conditioning network. That is, the calculations performed by the neural network can be conditioned or modulated by the coefficient vector. In some embodiments, conditioning the neural network may include a feature linear modulation FiLM. That is, a FiLM layer can be introduced into the architecture of the neural network, which is parameterized by conditioning based on the coefficient vector. For example, a randomly sampled conditioning vector λ can be fed into two multilayer perceptron (MLP) networks and vectors σ(λ) and μ(λ) with the same dimensions as the number of feature maps at the output of the modulated / conditioned convolution / transpose layer are created. Each feature map is first scaled by σ(λ). The scaled feature map is then offset by μ(λ).

[0067] Once the neural network is conditioned, in step S103, the conditioned neural network is trained based on the audio training signal. The training may involve calculating a loss function for the audio training signal processed by the conditioned neural network using weight coefficients indicated by the coefficient vector. In some embodiments, the training of the conditioned neural network may be performed in a perceptual weighted domain.

[0068] The described approach enables the neural network to learn to model the entire family of loss functions. The architecture and training of the neural network are described in further detail below.

[0069] Neural Network Architecture

[0070] It should be noted that the above method can be performed using any neural network, and therefore, the architecture of the neural network is not limited. However, in some embodiments, the neural network can implement a deep learning-based generator, which includes an encoder stage and a decoder stage, each stage including multiple layers with one or more filters in each layer, and the last layer of the encoder stage is mapped to a latent feature space.

[0071] exist Figure 3A non-limiting example of a simple architecture of a generator is schematically illustrated in the example of . The generator 1000 includes an encoder stage 1001 and a decoder stage 1002. The encoder stage 1001 and the decoder stage 1002 of the generator 1000 can be fully convolutional. The decoder stage 1002 can be a mirror image of the encoder stage 1001. The encoder stage 1001 and the decoder stage 1002 can each include multiple layers 1001a, 1001b, 1001c, 1002a, 1002b and 1002c with multiple filters in each layer, wherein the multiple filters in each layer of the decoder stage can perform filtering operations to produce multiple feature maps, and the last layer of the encoder stage 1001 can be mapped to a latent feature space representation c * , 1003.

[0072] That is, the encoder stage 1001 and the decoder stage 1002 may each include L layers, with N filters in each layer of the L layers. L may be a natural number ≥1, and N may be a natural number ≥1. Although the size of the N filters (also called the kernel size) is not limited, however, the size of the filters may be the same in each layer of the L layers, for example, the size of the filters may be 31. The number of filters may increase in each layer. For example, each filter may operate on an audio signal input to each layer of a generator with a stride of 2. Therefore, a learnable downsampling of a factor of 2 may be performed in the encoder layer, and a learnable upsampling of a factor of 2 may be performed in the decoder layer. In other words, the encoder stage 1001 of the generator may include multiple 1D-convolutional layers with a stride of 2 (without a bias term), and the decoder stage 1002 of the generator may include multiple 1D-transposed convolutional layers with a stride of 2 (without a bias term).

[0073] In some embodiments, conditioning the neural network may involve conditioning on one or more layers of the encoder stage of the generator adjacent to the latent feature space. In some embodiments, the conditioning may be FiLM conditioning as described above.

[0074] In at least one layer of the encoder stage 1001, a nonlinear operation may be additionally performed as an activation, the activation comprising one or more of a parametric rectified linear unit (PReLU), a rectified linear unit (ReLU), a leaky rectified linear unit (LReLU), an exponential linear unit (eLU), and a scaled exponential linear unit (SeLU). For example, the nonlinear operation may be based on a PReLU.

[0075] In an embodiment, the generator 1000 may further include a non-strided (stride=1) transposed convolution layer as an output layer after the last layer 1002a of the decoder stage 1002. For example, in the case of a mono audio signal, the output layer may include N=1 filters, and for example, for a stereo audio signal as an example of a multi-channel audio signal, the output layer may include N=2 filters. The size of the filter may be 31. At the output layer, the activation may be based on a tanh operation and a tanh(x) activation, since the audio signal output of the decoder stage 1002 needs to be restricted to +1 and -1.

[0076] like Figure 3 As illustrated in the example of , in an embodiment, one or more skip connections 1005 may exist between corresponding homologous layers of the encoder stage 1001 and the decoder stage 1002 of the generator 1000. In this case, the latent feature space representation c may be bypassed. * , 1003, to prevent information loss. The skip connection 1005 can be implemented using one or more concatenations and signal additions. Due to the implementation of the skip connection 1005, the number of filter outputs can be "virtually" doubled.

[0077] In an embodiment, a random noise vector z, 1004 may also be applied to the latent feature space representation c * , 1003, to modify the audio.

[0078] Discriminator Architecture

[0079] As described above, in some embodiments, the generator may be a generator trained in a generative adversarial network setting (GAN setting). A GAN setting typically includes a generator G and a discriminator D, which are trained through an iterative process. The architecture of the discriminator may have the same structure as the encoder stage of the generator. In other words, the architecture of the discriminator may be a mirror image of the structure of the encoder stage of the generator, however, without conditioning. That is, the discriminator may include multiple layers with multiple filters in each layer. That is, the discriminator may include L layers, each of the L layers having N filters in each layer. L may be a natural number ≥1, and N may be a natural number ≥1. Although the size of the N filters (also referred to as the kernel size) is unrestricted, the size of the filter may be the same in each of the L layers, for example, the size of the filter may be 31. The number of filters may increase in each layer. For example, each filter may operate on an audio signal input to each layer of the discriminator with a stride of 2. In other words, the discriminator may include multiple 1D-convolutional layers (without bias terms) with a stride of 2. The non-linear operation performed in at least one of the layers of the discriminator may include an LReLU. Prior to this, the discriminator may include an input layer. The input layer may be a non-strided convolutional layer (stride = 1 means non-strided). The discriminator may also include an output layer. The output layer may have N = 1 filters (the discriminator makes a single true / false decision) with a filter size of 1. In this case, the filter size of the output layer may be different from the filter size of the discriminator layer. Therefore, the output layer may be a one-dimensional (1D) convolutional layer that does not downsample the hidden activations. This means that the filters in the output layer can run with a stride of 1, while all previous layers of the discriminator can use a stride of 2. The activations in the output layer may be different from the activations of at least one of the discriminator layers. The activation may be sigmoid. However, if a least squares training method is used, a sigmoid activation may not be required, so the sigmoid activation is optional.

[0080] Loss-Conditioned Training Method for Generators in Generative Adversarial Network (GAN) Setting

[0081] In the following, the loss-conditioned training of the generator in the generative adversarial network setting (GAN setting) will be described. In general, during the training of the generative adversarial network setting, the generator G generates a processed audio training signal x by mapping to a latent feature space representation using an encoder stage and upsampling the latent feature space representation using a decoder stage. *. The audio training signals can be obtained from the encoded original audio signal x and the decoded original audio signal x, respectively. A random noise vector can be applied to the latent feature space representation. However, the random noise vector can be set to z=0. For coding artifact reduction, setting the random noise vector to z=0 can produce the best results. Alternatively, training can be performed without the input of the random noise vector z.

[0082] During training, the generator G attempts to output the processed audio training signal x * , the x * The generated processed audio training signal x * The original audio signal x is fed to the discriminator D one by one, and the discriminator D judges whether the input signal is the processed audio training signal x in a false / true manner. * Or it is the original audio signal x. In this case, the discriminator D tries to distinguish the original audio signal x from the processed audio training signal x. * During the iterative process, the generator G then adjusts its parameters to generate audio training signals x that are processed better and better than the original audio signal x * , and the discriminator D learns to better judge the processed audio training signal x * and the original audio signal x. This adversarial learning process can be described by the following formula (1):

[0083]

[0084] Note that the discriminator D can be trained first in order to train the generator G in the last step. Training and updating the discriminator D can involve assigning high scores to the original audio signal x and low scores to the processed audio training signal x. * The goal of training the discriminator D can be to identify the original audio signal (unencoded) as true, and the processed audio training signal x as false. * The (generated) identification is false. While training and updating the discriminator D, the parameters of the generator G can be kept fixed.

[0085] Training and updating the generator G may then involve comparing the original audio signal x with the generated processed audio training signal x * The goal of training the generator G can be to achieve the discriminator D to generate the processed audio training signal x * Recognized as true.

[0086] refer to Figure 2An example of, illustrating loss-conditioned training of a generator in a generative adversarial network (GAN) setting including a generator and a discriminator. The loss-conditioned training of the generator G, 100, may involve the following.

[0087] The coefficient vector λ is randomly sampled from the distribution p(λ) of the coefficients. The distribution can be uniform in the range [0,100]. It can be said that the coefficient vector represents an ordered group of weight coefficients λ of the corresponding loss function. The number of weight coefficients in the group is determined by the number and / or weight of each item in the corresponding loss function. Therefore, it can be said that the distribution of the coefficients represents the distribution of the loss function. In one example, the elements of the coefficient vector can be independently sampled from a predetermined range (e.g., the range [0,100]).

[0088] Once the coefficient vector is sampled, the generator G, 100 is conditioned based on the coefficient vector 108. The conditioning may include a feature linear modulation FiLM. In some embodiments, conditioning the neural network may involve conditioning on one or more layers of the encoder stage of the generator adjacent to the latent feature space. In other words, for example, only the one or more layers of the encoder stage may be conditioned by FiLM conditioning as described above.

[0089] The audio training signal can be 103 and an optional random noise vector z, 104 are input into the (FiLM) conditioned generator G, 100. In one embodiment, the random noise vector z may be set to z = 0. Alternatively, training may be performed without the input of the random noise vector z.

[0090] Audio training signal 103 can be obtained by encoding and decoding the original audio signal x, 102. Then, based on the input, the conditioned generator G, 100 generates the processed audio training signal x by mapping to the latent feature space representation using the encoder stage and upsampling the latent feature space representation using the decoder stage. * , 105.

[0091] One by one, we will get the audio training signal from it 103 original audio signal x, 102 and generated processed audio training signal x * , 105 is input to the discriminator D, 101. As additional information, the audio training signal can also be 103 is input to the discriminator D, 101. The discriminator D, 101 then determines 106 that the input data is the processed audio training signal x * , 105 (false) or the original audio signal x, 102 (true).

[0092] In the next step, the parameters of the conditioned generator G,100 are then adjusted until the discriminator D,101 is no longer able to distinguish the processed audio training signal x * , 105 with the original audio signal x, 102. This can be done in an iterative process 107.

[0093] The determination made by the discriminator D, 100 may be based on one or more perceptually motivated objective functions, such as those shown in the following formula (2):

[0094]

[0095] From the first term in formula (2), we can see that by A conditional generative adversarial network setting is applied by feeding the discriminator as additional information.

[0096] The two terms including coefficients λ2 and λ3 in the above (generator loss function) formula (2) can be called multi-resolution STFT loss terms. The multi-resolution STFT loss can be said to be the sum of different STFT-based loss functions using different STFT parameters. (spectral convergence loss) and (Log-scaled STFT magnitude loss) can apply STFT-based loss at M different resolutions with number of FFT bins ∈ {512, 1024, 2048}, hop size ∈ {50, 120, 240}, and final window length ∈ {240, 600, 1200}. The results show that for processing general audio (i.e. any content type), the multi-resolution STFT loss term provides quality improvement.

[0097] The term including the coefficient λ1 in formula (2) is the 1-norm distance scaled by the lambda factor λ1. The value of this lambda can be selected from 1 to 100, depending on the application and / or the length of the signal input to the generator. For example, λ1 can be selected as λ1=100. In addition, the scaling of the multi-resolution STFT loss term can be set to the same value as lambda.

[0098] item It also indicates a random sampling of the coefficient vector λ from the distribution p(λ). In the case of a generator loss function based on a multi-resolution STFT, the elements of the coefficient vector thus indicate the weight coefficients λ1 to λ3.

[0099] Alternatively, the generator loss function of formula (2) can be applied to the following form:

[0100]

[0101] In the above formula, the term is also used to cover the adversarial term (i.e., the first term on the right side of formula (2)). In addition, formula (2a) therefore also includes a weighted term λ0 in front of the adversarial term. In general, any term of the loss function can have an associated weighted term (weight coefficient), where the weight term is indicated by a coefficient vector λ, which can be said to represent a set of weight coefficients (ordered group). It is understood that the coefficient vector λ is selected to have an appropriate dimension to indicate all weighted terms of the envisaged loss function.

[0102] In some embodiments, training the conditioned neural network (conditioned generator) may be performed in the perceptually weighted domain. The perceptually weighted audio training signal may then be Input to the conditioned generator G, 100. Perceptually weighted audio training signal It can be obtained by encoding and decoding the perceptually weighted original audio signal x, which is obtained by applying a mask or mask curve P to the original audio signal, the mask or mask curve indicating the mask threshold obtained from the psychoacoustic model. Then, based on the input, the conditioned generator G, 100 generates a processed perceptually weighted audio training signal x * In this case, the perceptually weighted audio training signal is obtained from The perceptually weighted original audio signal x and the generated processed perceptually weighted audio training signal x * , are input into the discriminator D,101.

[0103] In some embodiments, randomly sampling coefficient vectors, conditioning the neural network, and training the conditioned neural network may form at least a portion of an epoch, and the method may further include performing two or more epochs for each audio content type in a set of audio content types. The set of audio content types may include, for example, one or more of speech, music, speech and music, and applause. When performing two or more epochs for each audio content type in a set of audio content types, the neural network may be trained with multiple conditions. During multi-condition training, the neural network "sees" the distribution of weight coefficients and "selects" the best coefficient vector that works best across multiple conditions. It should be noted that in addition to audio content type, the multiple conditions may also include one or more of a set of bit rates and a set of codecs.

[0104] For example, in each epoch, an original audio signal is obtained from a set of audio samples associated with the content type. Then, a coefficient vector is randomly sampled and the neural network is conditioned (generator) using the coefficient vector. Then, the conditioned neural network processes the corresponding audio training signal and calculates a loss function for the processed audio training signal, wherein the weight coefficient of the loss function is given by the coefficient vector λ. Then, training is continued until the value of the calculated loss function is minimized as described above. That is, in each epoch, for each pair of the audio training signal obtained from the original audio signal obtained from a set of audio samples and the randomly sampled coefficient vector, the parameters of the conditioned generator are tuned until the discriminator can no longer distinguish the processed audio training signal from the original audio signal. When two or more epochs are performed for each audio content type in a set of audio content types, a single model can be trained to cover different signal categories. However, for each signal category, training produces a best possible coefficient vector. In a similar manner, one or more epochs can also be performed for a set of bit rates and / or codecs. For example, the neural network can first be trained on a set of audio content types as described above. The same neural network can then be further trained on a set of bitrates and / or a set of codecs, or vice versa. This training produces a single coefficient vector that best fits the multiple conditions experienced during training. However, this training approach also enables a single neural network during inference to "remember" which coefficient vector produced the best results during training for a particular condition (bitrate / codec / content type). During inference, that vector can then be selected / included in the conditioning information based on the corresponding bitrate / content type / codec.

[0105] Reference again Figure 2 In general, the training of the discriminator D, 101 can follow the same general process as the training of the generator G, 100 described above, except that in this case, the parameters of the generator G, 100 can be fixed, while the parameters of the discriminator D, 101 can be varied. The training of the discriminator D, 101 can be described by the following formula (3), which enables the discriminator D, 101 to transform the processed audio training signal x * , 105 is determined to be false:

[0106]

[0107] In the above case, the audio training signal is also As additional information is input to the discriminator D, 101 applies the least squares method (LS) and the conditional generative adversarial network setting.

[0108] In addition to the least squares method, other training methods can be used to train the conditioned generator in the generative adversarial network setting. The present disclosure is not limited to a specific training method. Alternatively, or additionally, the so-called Wasserstein method can also be used. In this case, the Earth Mover Distance, also called the Wasserstein Distance, can be used instead of the least squares distance. In general, different training methods will make the training of the discriminator and the conditioned generator more stable. However, the type of training method applied does not affect the architecture of the (conditioned) generator.

[0109] Method for processing audio signals using neural networks trained with loss conditions

[0110] refer to Figure 4 An example of a computer-implemented method for processing an audio signal using a neural network trained with loss conditions is illustrated. In step S201, the neural network is conditioned based on conditioning information including a coefficient vector. In some embodiments, as described above, the elements of the coefficient vector may indicate weight coefficients of a loss function. In some embodiments, the loss function may be a multi-objective loss function, such as the generator loss function based on a multi-resolution STFT of formula (2).

[0111] In some embodiments, the conditioning information may be based on (determined by) the content type and / or bitrate of the audio signal to be processed. The conditioning information may be further selected depending on the training results, for example, based on which coefficient vector works best. As described above, the loss-conditioned trained neural network may have been trained by performing two or more epochs on each of a set of audio content types so that the neural network can cover a variety of different categories and conditions. In a similar manner, this may apply to bitrate and / or codec. That is, the conditioning information may alternatively or additionally also be based on the codec.

[0112] In some embodiments, conditioning the neural network may include a feature linear modulation FiLM as described above. In some embodiments, the neural network may implement a deep learning based generator. The generator may include an encoder stage and a decoder stage, each stage including multiple layers with one or more filters in each layer, and the last layer of the encoder stage maps to a latent feature space. Then, conditioning the neural network may involve conditioning on one or more layers of the encoder stage of the generator adjacent to the latent feature space.

[0113] Additionally, in some embodiments, a random noise vector z may be applied to the latent feature space to modify the audio.

[0114] refer to Figure 5 In some embodiments, the method may further include receiving an audio bitstream 200, the audio bitstream 200 including an audio signal and conditioning information. In this case, the method may further include core decoding the audio bitstream, for example, by an audio decoder 201, to obtain an audio signal 203. And the method may include extracting conditioning information 204 from the received bitstream 200. In an exemplary embodiment, the audio bitstream may include metadata, and the conditioning information may be included in the metadata, or the metadata may indicate the conditioning information. The trained neural network 202 may then be conditioned based on the extracted conditioning information 204. The audio signal 203 and an optional random noise vector z 205 may then be input into the trained and conditioned neural network 202 to process the audio signal 203. Then, a processed audio signal 206 is obtained as an output of the trained and conditioned neural network 202.

[0115] Alternatively, in some embodiments, the method may further include analyzing the audio signal 203 and determining the conditioning information 204 based on a result of the analysis.

[0116] Reference again Figure 4 In the example of, in step S202, the audio signal is input into the conditioned neural network to process the audio signal. In step S203, the conditioned neural network then processes the audio signal based on the conditioning information. And, in step S204, as the output of the conditioned neural network, the processed audio signal is then obtained.

[0117] In some embodiments, the method may be performed in a perceptually weighted domain. Then, a processed audio signal in the perceptually weighted domain may be obtained as an output of the conditioned neural network. In this case, the method may also include converting the processed audio signal from the perceptually weighted domain to the original signal domain. In some embodiments, the neural network may also be trained in the perceptually weighted domain.

[0118] refer to Figure 6 For example, the above method can be implemented by a device for processing an audio signal using a neural network trained using a loss condition. The device 300 may include one or more processors 301 configured to execute the above method.

[0119] Alternatively, or additionally, the methods described herein may also be implemented by a computer program comprising instructions which, when executed by a computing device, cause the computing device to perform the methods as described herein. The computer program may be provided on a computer-readable storage medium storing the computer program.

[0120] Results show that a neural network trained as described herein is able to generate outputs (processed audio signals) for different content types / bitrates / codecs that are as good as outputs generated by a separate neural network trained on the corresponding specific task. That is, the methods and apparatus described herein allow a single neural network to be trained and inferred for tasks that would normally require a large number of separately trained neural networks. This one-to-many approach provides a simple way to improve the efficiency of training and inference. Specifically, because only one neural network needs to be stored, the method reduces memory consumption on the decoder side.

[0121] explain

[0122] Unless otherwise expressly stated, as will be apparent from the following discussion, it should be understood that, throughout the discussion of this disclosure, the use of terms such as "process," "compute," "determine," and "analyze" refer to the actions and / or processes of a computer or computing system or similar electronic computing device that manipulate and / or transform data represented as physical (e.g., electronic) quantities into other data similarly represented as physical quantities.

[0123] In a similar manner, the term "processor" may refer to any device or any portion of a device that processes electronic data, e.g., from registers and / or memory, e.g., transforms that electronic data into other electronic data, e.g., that may be stored in registers and / or memory. A "computer" or "computing machine" or "computing platform" may include one or more processors.

[0124] In an exemplary embodiment, the methods described herein may be performed by one or more processors that accept computer-readable (also referred to as machine-readable) codes including a set of instructions that, when executed by one or more processors, perform at least one of the methods described herein. Any processor capable of executing a set of instructions (sequential or otherwise) for specifying an action to be taken is included. Thus, an example is a typical processing system including one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may also include a memory subsystem including a main RAM and / or static RAM, and / or a ROM. A bus subsystem for communication between components may be included. The processing system may also be a distributed processing system having processors coupled via a network. If the processing system requires a display, such a display may be included, for example, a liquid crystal display (LCD) or a cathode ray tube (CRT) display. If manual data entry is required, the processing system also includes an input device, such as one or more alphanumeric input units, such as a keyboard and a pointing control device, such as a mouse, and the like. The processing system may also include a storage system such as a disk drive unit. In some configurations, the processing system may include a sound output device and a network interface device. Therefore, the memory subsystem includes a computer-readable carrier medium that carries a computer-readable code (e.g., software) including a set of instructions that, when executed by one or more processors, cause one or more methods described herein to be performed. Note that when a method includes several elements, such as several steps, the order of these elements is not implied unless otherwise specified. The software may reside in a hard disk, or may reside completely or at least partially in a RAM and / or a processor during execution of the software by a computer system. Therefore, the memory and the processor also constitute a computer-readable carrier medium that carries a computer-readable code. In addition, the computer-readable carrier medium may form or be included in a computer program product.

[0125] In alternative exemplary embodiments, one or more processors operate as standalone devices or may be connected, for example, in a networked deployment, to other processor(s), one or more processors may operate as a server or user machine in a server-user network environment, or as a peer machine in a peer-to-peer or distributed network environment. The one or more processors may form a personal computer (PC), a tablet PC, a personal digital assistant (PDA), a cellular phone, a network appliance, a network router, a switch or a bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by the machine.

[0126] Note that the term "machine" shall also include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

[0127] Therefore, an exemplary embodiment of each method described herein exists in the form of a computer-readable carrier medium carrying a set of instructions, for example, a computer program for execution on one or more processors, for example, one or more processors as part of a network server arrangement. Therefore, as will be appreciated by those skilled in the art, exemplary embodiments of the present disclosure may be embodied as a method, a device such as a special-purpose device, a device such as a data processing system, or a computer-readable carrier medium, for example, a computer program product. The computer-readable carrier medium carries a computer-readable code including a set of instructions, which, when executed on one or more processors, causes the processor or multiple processors to implement the method. Therefore, various aspects of the present disclosure may take the form of a method, a fully hardware exemplary embodiment, a fully software exemplary embodiment, or an exemplary embodiment combining software and hardware aspects. In addition, the present disclosure may take the form of a carrier medium (e.g., a computer program product on a computer-readable storage medium) to carry the computer-readable program code embodied in the medium.

[0128] The software may also be transmitted or received over a network via a network interface device. When the carrier medium is a single medium in an exemplary embodiment, the term "carrier medium" shall be understood to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) storing one or more sets of instructions. The term "carrier medium" shall also be understood to include any medium capable of storing, encoding, or carrying a set of instructions for execution by one or more processors, and causing one or more processors to execute any one or more methods of the present disclosure. The carrier medium may take a variety of forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical disks, magnetic disks, and magneto-optical disks. Volatile media include dynamic memory, such as main memory. Transmission media include coaxial cables, copper wires, and optical fibers, including wires that constitute a bus subsystem. Transmission media may also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communications. For example, the term "carrier medium" should be understood accordingly to include, but is not limited to, solid-state memories, computer products embodied in optical and magnetic media; media that carry propagated signals detectable by at least one processor or one or more processors and that represent a set of instructions for implementing a method when executed; and transmission media in a network that carries propagated signals detectable by at least one of one or more processors and that represent a set of instructions.

[0129] It will be appreciated that the steps of the method discussed are performed in an exemplary embodiment by a suitable processor (or processors) of a processing (e.g., computer) system executing instructions (computer readable code) stored in a storage device. It will also be appreciated that the present disclosure is not limited to any particular implementation or programming technique, and that the present disclosure may be implemented using any suitable technique for implementing the functionality described herein. The present disclosure is not limited to any particular programming language or operating system.

[0130] References throughout this disclosure to "one embodiment," "some embodiments," or "an exemplary embodiment" mean that a particular feature, structure, or characteristic associated with that embodiment is included in at least one embodiment of the present disclosure. Thus, the appearance of the phrases "in one embodiment," "in some embodiments," or "in an exemplary embodiment" in various places throughout this disclosure are not necessarily all referring to the same exemplary embodiment. Furthermore, it will be apparent to one of ordinary skill in the art of this disclosure that the particular features, structures, or characteristics may be combined in any suitable manner in one or more exemplary embodiments.

[0131] As used herein, unless otherwise indicated, the use of ordinal adjectives "first," "second," "third," etc. to describe common objects merely indicates different instances of similar objects being referenced and is not intended to imply that the objects so described must have a given order in time, space, ranking, or in any other manner.

[0132] In the following claims and the specification herein, any of the terms comprising, including or which comprises is an open term that means including at least the elements / features that follow, but does not exclude other elements / features. Therefore, when the term comprising is used in the claims, it should not be interpreted as being limited to the components or elements or steps listed thereafter. For example, the scope of the device that states that it includes A and B should not be limited to a device consisting of only elements A and B. Any of the terms including or which includes or that includes used herein is also an open term that means including at least the elements / features that follow the term, but does not exclude other elements / features. Therefore, including is synonymous with comprising and means comprising.

[0133] It should be understood that in the above description of exemplary embodiments of the present disclosure, various features of the present disclosure are sometimes combined in a single exemplary embodiment, figure, or description thereof for the purpose of streamlining the present disclosure and aiding in the understanding of one or more of the various inventive aspects. However, this method of disclosure should not be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. On the contrary, as reflected in the following claims, inventive aspects reside in fewer features than all of the features of a single, foregoing disclosed exemplary embodiment. Therefore, the claims following the specification are hereby expressly incorporated into this specification, with each claim standing alone as a separate exemplary embodiment of the present disclosure.

[0134] In addition, as will be appreciated by those skilled in the art, although some exemplary embodiments described herein include some features that are included in other exemplary embodiments but not others, the combination of features of different exemplary embodiments is intended to be within the scope of the present disclosure and form different exemplary embodiments. For example, in the claims below, any of the claimed exemplary embodiments may be used in any combination.

[0135] In the description provided herein, many specific details are set forth. However, it should be understood that exemplary embodiments of the present disclosure may be implemented without these specific details. In other examples, well-known methods, structures, and techniques are not shown in detail in order not to obscure the understanding of this description.

[0136] Therefore, while what is believed to be the best mode of the present disclosure has been described, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of the present disclosure, and it is intended that all such changes and modifications falling within the scope of the present disclosure be claimed. For example, any formula given above is merely representative of a procedure that may be used. Functions may be added or deleted from the block diagrams, and operations may be interchanged between functional blocks. Steps may be added or deleted to the methods described within the scope of the present disclosure.

[0137] Various aspects and implementations of the present disclosure may also be understood from the following enumerated exemplary embodiments (EEE), which are not claims.

[0138] EEE 1. A method for loss condition training of a neural network, the method comprising:

[0139] Randomly sampling a coefficient vector from a distribution of coefficients, wherein elements of the coefficient vector indicate weight coefficients of a loss function;

[0140] conditioning the neural network based on the coefficient vector; and

[0141] A conditioned neural network is trained based on an audio training signal, wherein the training involves calculating the loss function for the audio training signal processed by the conditioned neural network using the weight coefficients indicated by the coefficient vector.

[0142] EEE 2. The method according to EEE 1, wherein the loss function is a multi-objective loss function.

[0143] EEE 3. The method according to EEE 1 or 2, wherein the distribution of the coefficients is a uniform distribution within a predetermined range.

[0144] EEE 4. A method according to any one of EEEs 1 to 3, wherein conditioning the neural network comprises feature linear modulation (FiLM).

[0145] EEE 5. A method according to any one of EEEs 1 to 4, wherein the coefficient vector is randomly sampled, the neural network is conditioned, and the conditioned neural network is trained to form at least a portion of a period, and wherein the method further includes performing two or more periods for each audio content type in a set of audio content types.

[0146] EEE 6. The method according to any one of EEE 1 to 5, wherein training the conditioned neural network is performed in the perceptual weighted domain.

[0147] EEE 7. A method according to any one of EEEs 1 to 6, wherein the neural network implements a deep learning-based generator, the generator comprising an encoder stage and a decoder stage, each stage comprising multiple layers with one or more filters in each layer, and the last layer of the encoder stage is mapped to a latent feature space.

[0148] EEE 8. A method according to EEE 6, wherein the generator is trained in a generative adversarial network (GAN) setting including the generator and the discriminator.

[0149] EEE 9. A method according to EEE 7 or 8, wherein conditioning the neural network involves conditioning on one or more layers of the encoder stage of the generator adjacent to the latent feature space.

[0150] EEE 10. The method according to EEE 8 or 9, wherein training the conditioned neural network comprises:

[0151] Input the audio training signal into the conditioned generator;

[0152] generating, by the conditioned generator, a processed audio training signal based on the audio training signal;

[0153] inputting the processed audio training signal and the corresponding original audio signal from which the audio training signal is obtained into the discriminator one by one;

[0154] determining, by the discriminator, whether the input audio signal is the processed audio training signal or the original audio signal; and

[0155] The parameters of the generator are iteratively adjusted until the discriminator can no longer distinguish between the processed audio training signal and the original audio signal.

[0156] EEE 11. The method according to EEE 10, wherein a random noise vector z is also applied to the latent feature space to modify the audio.

[0157] EEE 12. A computer-implemented method for processing an audio signal using a neural network trained with a loss condition, the method comprising:

[0158] conditioning the neural network based on conditioning information including a coefficient vector;

[0159] Inputting the audio signal into a conditioned neural network to process the audio signal;

[0160] processing, by the conditioned neural network, the audio signal based on the conditioning information; and

[0161] The processed audio signal is obtained as the output of the conditioned neural network.

[0162] EEE 13. The method according to EEE 12, wherein the elements of the coefficient vector indicate weight coefficients of the loss function.

[0163] EEE 14. The method according to EEE 13, wherein the loss function is a multi-objective loss function.

[0164] EEE 15. The method according to any one of EEE 12 to 14, wherein the conditioning information is based on a content type and / or a bit rate of the audio signal. ...

[0165] EEE 16. A method according to any one of EEE 12 to 15, wherein conditioning the neural network comprises feature linear modulation FiLM.

[0166] EEE 17. A method according to any one of EEE 12 to 16, wherein the neural network implements a deep learning based generator, the generator comprising an encoder stage and a decoder stage, each stage comprising multiple layers with one or more filters in each layer, and the last layer of the encoder stage is mapped to a latent feature space.

[0167] EEE 18. A method according to EEE 17, wherein conditioning the neural network involves conditioning on one or more layers of the encoder stage of the generator that are adjacent to the latent feature space.

[0168] EEE 19. A method according to EEE 18 or 19, wherein a random noise vector z is applied to the latent feature space to modify the audio.

[0169] EEE 20. The method according to any one of EEE 12 to 19, wherein the method further comprises receiving an audio bitstream comprising the audio signal and the conditioning information.

[0170] EEE 21. The method according to EEE 20, wherein the method further comprises a core decoding the audio bit stream to obtain the audio signal.

[0171] EEE 22. The method according to EEE 20 or 21, wherein the method further comprises extracting the conditioning information from the received bit stream.

[0172] EEE 23. The method according to any one of EEEs 12 to 21, wherein the method further comprises analyzing the audio signal and determining the conditioning information based on a result of the analysis. ...

[0173] EEE 24. A method according to any one of EEE 12 to 23, wherein the method is performed in a perceptually weighted domain, and wherein a processed audio signal in the perceptually weighted domain is obtained as an output of the conditioned neural network.

[0174] EEE 25. The method according to EEE 24, wherein the method further comprises converting the processed audio signal from the perceptually weighted domain to the original signal domain.

[0175] EEE 26. A method according to any one of EEEs 12 to 25, wherein the neural network has been trained in the perceptual weighted domain.

[0176] EEE 27. An apparatus for processing an audio signal using a neural network trained with a loss condition, the apparatus comprising one or more processors configured to perform a method comprising:

[0177] conditioning the neural network based on conditioning information including a coefficient vector;

[0178] Inputting the audio signal into a conditioned neural network to process the audio signal;

[0179] processing, by the conditioned neural network, the audio signal based on the conditioning information; and

[0180] The processed audio signal is obtained as the output of the conditioned neural network.

[0181] EEE 28. A computer program comprising instructions which, when executed by a computing device, cause the computing device to perform a method according to any one of EEE 1 to 11.

[0182] EEE 29. A computer-readable storage medium storing the computer program according to EEE 28.

[0183] EEE 30. A computer program comprising instructions which, when executed by a computing device, cause the computing device to perform a method according to any one of EEE 12 to 26. ...

[0184] EEE 31. A computer-readable storage medium storing the computer program according to EEE 30.

Claims

1. A computer-implemented method for loss-conditioned training of a neural network that outputs an enhanced audio signal, the method comprising: Randomly sampling a coefficient vector from a distribution of coefficients, wherein elements of the coefficient vector indicate weight coefficients corresponding to loss terms of a loss function; conditioning the neural network based on the coefficient vector; and A conditioned neural network is trained based on an audio training signal, wherein the training involves calculating the loss function for the audio training signal processed by the conditioned neural network using the weight coefficients indicated by the coefficient vector.

2. The method according to claim 1, wherein: The loss function is a multi-objective loss function.

3. The method according to claim 1 or 2, wherein: The distribution of the coefficients is a uniform distribution within a predetermined range.

4. The method according to any one of claims 1 to 3, wherein: Conditioning the neural network includes feature linear modulation FiLM.

5. The method according to any one of claims 1 to 4, wherein: Randomly sampling the coefficient vector, conditioning the neural network, and training the conditioned neural network form at least a portion of an epoch, and wherein the method further comprises performing two or more epochs for each audio content type in a set of audio content types.

6. The method according to any one of claims 1 to 5, wherein: Training the conditioned neural network is performed in the perceptually weighted domain.

7. The method according to any one of claims 1 to 6, wherein: The neural network implements a deep learning based generator, the generator comprising an encoder stage and a decoder stage, each stage comprising multiple layers with one or more filters in each layer, the last layer of the encoder stage being mapped to a latent feature space.

8. The method according to claim 7, wherein: Conditioning the neural network involves conditioning on one or more layers of the encoder stage of the generator that are adjacent to the latent feature space.

9. The method according to claim 7 or 8, wherein: The generator is trained in a generative adversarial network (GAN) setting including the generator and a discriminator.

10. The method according to claim 9, wherein: Training the conditioned neural network includes: Input the audio training signal into the conditioned generator; generating, by the conditioned generator, a processed audio training signal based on the audio training signal; inputting the processed audio training signal and the corresponding original audio signal from which the audio training signal is obtained into the discriminator one by one; determining, by the discriminator, whether the input audio signal is the processed audio training signal or the original audio signal; and The parameters of the generator are iteratively adjusted until the discriminator can no longer distinguish between the processed audio training signal and the original audio signal.

11. The method according to claim 10, wherein: A random noise vector z is also applied to the latent feature space to modify the audio.

12. A computer-implemented method for processing an audio signal using a neural network trained with a loss condition, the method comprising: conditioning the neural network based on conditioning information including a coefficient vector, wherein elements of the coefficient vector indicate weight coefficients corresponding to loss terms of a loss function; Inputting the audio signal into a conditioned neural network to process the audio signal; processing, by the conditioned neural network, the audio signal based on the conditioning information; and An enhanced audio signal is obtained as an output of the conditioned neural network.

13. The method according to claim 12, wherein: The loss function is a multi-objective loss function.

14. The method according to claim 12 or 13, wherein: The conditioning information is based on a content type and / or a bit rate of the audio signal.

15. The method according to any one of claims 12 to 14, wherein: Conditioning the neural network includes feature linear modulation FiLM.

16. The method according to any one of claims 12 to 15, wherein: The neural network implements a deep learning based generator, the generator comprising an encoder stage and a decoder stage, each stage comprising multiple layers with one or more filters in each layer, the last layer of the encoder stage being mapped to a latent feature space.

17. The method according to claim 16, wherein: Conditioning the neural network involves conditioning on one or more layers of the encoder stage of the generator that are adjacent to the latent feature space.

18. The method according to claim 16 or 17, wherein: A random noise vector z is applied to the latent feature space to modify the audio.

19. The method according to any one of claims 12 to 18, wherein: The method further comprises receiving an audio bitstream comprising the audio signal and the conditioning information.

20. The method according to claim 19, wherein: The method also includes a core decoding the audio bitstream to obtain the audio signal.

21. The method according to claim 19 or 20, wherein: The method further comprises extracting the conditioning information from the received bitstream.

22. The method according to any one of claims 12 to 20, wherein: The method further comprises analyzing the audio signal and determining the conditioning information based on a result of the analysis.

23. The method according to any one of claims 12 to 22, wherein: The method is performed in a perceptually weighted domain and wherein an enhanced audio signal is obtained in the perceptually weighted domain as an output of the conditioned neural network.

24. The method according to claim 23, wherein: The method further comprises converting the enhanced audio signal from the perceptually weighted domain to an original signal domain.

25. The method according to any one of claims 12 to 24, wherein: The neural network has been trained in the perceptually weighted domain.

26. An apparatus for processing an audio signal using a neural network trained with a loss condition, the apparatus comprising one or more processors configured to perform a method comprising: The neural network is conditioned based on conditioning information including a coefficient vector, wherein The elements of the coefficient vector indicate weight coefficients corresponding to loss terms of the loss function; Inputting the audio signal into a conditioned neural network to process the audio signal; processing the audio signal by the conditioned neural network based on the conditioning information; as well as An enhanced audio signal is obtained as an output of the conditioned neural network.

27. A computer program comprising instructions which, when executed by a computing device, cause the computing device to perform the method according to any one of claims 1 to 11.

28. A computer-readable storage medium storing the computer program according to claim 27.

29. A computer program comprising instructions which, when executed by a computing device, cause the computing device to perform a method according to any one of claims 12 to 25.

30. A computer-readable storage medium storing the computer program according to claim 29.

Images