Generation of multi-channel audio signal and data signal representing multi-channel audio signal

Abstract

An audio apparatus comprises a receiver (101) arranged to receive a data signal comprising a downmix audio signal for a multi-channel audio signal, upmix parameterized data for upmixing the downmix audio signal, and a set of control data values. An artificial neural network (107) has an input node for receiving a second sample of the downmix audio signal and a node for receiving control data values from a set of control data values. Based on these inputs, an artificial neural network (107) generates samples of an auxiliary audio signal for the downmix audio signal. A generator (105) generates a multi-channel audio signal from the downmix signal and the auxiliary signal in accordance with the upmix parameterized data. Another device may generate a set of control data values using another artificial neural network having input nodes for receiving downmix of the multi-channel audio signal. In many embodiments, the operation may be sub-band based, with separate artificial neural networks for different sub-bands.

Description

Technical Field

[0001] The present invention relates to the generation of a multi-channel audio signal and / or a data signal representing a multi-channel audio signal, and in particular, but not exclusively, to the encoding and / or decoding of a stereo signal. Background Art

[0002] Spatial audio applications have become numerous and widespread and increasingly form at least a part of many audiovisual experiences. Indeed, new and improved spatial experiences and applications are constantly being developed, which results in increased demands on audio processing and rendering.

[0003] For example, in recent years, virtual reality (VR) and augmented reality (AR) have received increasing interest and several implementations and applications are reaching the consumer market. In fact, devices are being developed for rendering experiences and for capturing or recording suitable data for such applications. For example, relatively low-cost devices are being developed to allow game consoles to provide a complete VR experience. It is expected that this trend will continue, and in fact will increase as the markets for VR and AR reach considerable size in a short time frame. In the audio domain, a prominent area explores the reproduction and synthesis of realistic and natural spatial audio. The ideal goal is to generate natural audio sources so that users cannot recognize the difference between synthesized or original audio sources.

[0004] A great deal of research and development effort has been focused on providing efficient and high quality audio encoding and audio decoding for spatial audio. Frequently used spatial audio representations are multi-channel audio representations, including stereo representations, and efficient encoding of such multi-channel audio has been developed based on downmixing the multi-channel audio signal to a downmix channel with fewer channels. One of the major advances in low bitrate audio coding is the use of parametric multi-channel coding, where the downmix signal is generated together with parametric data that can be used to upmix the downmix signal to recreate the multi-channel audio signal.

[0005] In particular, instead of conventional mid-side or intensity decoding, in parametric multi-channel audio decoding, the multi-channel input signal is downmixed to a lower number of channels (e.g., two to one), and the multi-channel image (stereo) parameters are extracted. The downmix signal is then encoded using a more conventional audio decoder (e.g., a mono audio encoder). The downmixed bitstream is multiplexed with an encoded multi-channel image parameter bitstream. The bitstream is then sent to a decoder, where the process is reversed. First, the downmix audio signal is decoded, after which the multi-channel audio signal is reconstructed guided by the encoded multi-channel image / upmix parameters.

[0006] An example of stereo decoding is described in E. Schuijers, W. Oomen, B. den Brinker, J. Breebaart, "Advances in Parametric Coding for High-Quality Audio", 114th AES Convention, Amsterdam, The Netherlands, 2003, Preprint 5852. In the described method, the downmixed mono signal is parameterized by exploiting the natural separation of the signal into three components (objects): transients, sinusoids and noise. More details are provided in "Low Complexity Parametric Stereo Coding", 116th AES, Berlin, Germany, 2004, Preprint 6073, which describes how parametric stereo can be implemented with low (decoder) complexity when combining parametric stereo with spectral band replication (SBR).

[0007] In the described method, decoding is based on the use of a so-called decorrelation process. The decorrelation process generates a decorrelation helper signal from the mono signal. In the stereo reconstruction process, both the mono signal and the decorrelation helper signal are used to generate an upmixed stereo signal based on the upmix parameters. Specifically, the two signals can be multiplied by a 2x2 matrix related to time and frequency with coefficients determined from the upmix parameters to provide an output stereo signal.

[0008] However, although parametric stereo (PS) and similar downmix encoding / decoding methods are a jump from traditional stereo and multi-channel decoding, the method is not optimal in all scenarios. In particular, known encoding and decoding methods tend to introduce some distortions, changes, artifacts, etc., which may introduce differences between the (original) multi-channel audio signal input to the encoder and the multi-channel audio signal recreated at the decoder. Typically, the audio quality may degrade and imperfect reconstruction of the multi-channel occurs. In addition, the data rate may still be higher than expected, and / or the complexity of the processing / resource usage may be higher than preferred.

[0009] A further problem is that it is particularly desirable to reduce the complexity and computational burden on the decoder side, even if this increases the complexity and computational resources on the encoder side. For example, in an audio distribution system, such as a system that provides audiovisual content from a central server to many clients, encoding is performed only once, while decoding may be performed multiple times in separate decoding devices, which may typically have fewer computational resources than the server device.

[0010] Furthermore, it is often desirable that the encoder side be able to modify or control the generation of the audio signal at the decoder side. It is desirable that the encoder can be arranged to adapt the operation of the decoder, for example to provide desired audio characteristics, or to allow a debugging process to result in a more accurate reconstruction of the multi-channel audio signal at the decoder side.

[0011] Therefore, improved methods would be advantageous. In particular, methods that allow increased flexibility, improved adaptability, improved performance, increased audio quality, improved audio quality vs. data rate tradeoffs, reduced complexity and / or resource usage, improved remote control of audio processing, improved encoder-side inputs on decoder-side operations / processing, improved distribution of computational resources between encoder-side and decoder-side processing, reduced computational load, facilitated implementation, and / or improved spatial audio experience would be advantageous. Summary of the invention

[0012] Accordingly, the Invention seeks to preferably mitigate, alleviate or eliminate one or more of the above mentioned disadvantages singly or in any combination.

[0013] According to an aspect of the present invention, there is provided an apparatus for generating a multi-channel audio signal, the apparatus comprising: a receiver (101) for receiving a data signal, the data signal comprising: a downmix audio signal for the multi-channel signal; upmix parameterization data for upmixing the downmix audio signal; a set of control data values ​​for adapting processing to generate an auxiliary signal for upmixing from the downmix audio signal; an artificial neural network (107) having an input node for receiving samples of the downmix audio signal and an output node arranged to provide samples of the auxiliary audio signal, the artificial neural network (107) further comprising a node arranged to include a contribution from the set of control data values; and a generator (105) arranged to generate the multi-channel audio signal from the downmix signal and the auxiliary signal according to the upmix parameterization data.

[0014] In many embodiments, the method may provide an improved audio experience.For many signals and scenarios, the method may provide an improved generation / reconstruction of a multi-channel audio signal with improved perceived audio quality.

[0015] The method may provide a particularly advantageous arrangement which in many embodiments and scenarios may allow the possibility of facilitating and / or improving the use of artificial neural networks in audio processing (typically including audio encoding and / or decoding). The method may allow the advantageous use of artificial neural networks when generating a multi-channel audio signal from a downmix audio signal.

[0016] In many embodiments, the method may allow for the generation of an improved multi-channel audio signal by allowing an encoder to adapt / modify the processing by which the device generates the multi-channel audio signal. The method may enable or facilitate such remote control by providing a method that is compatible with and exploits the advantages that may be achieved by employing an artificial neural network as part of the audio processing.

[0017] The method may provide an efficient implementation and in many embodiments may allow reduced complexity and / or resource usage. The method may in many cases allow a data rate reduction for data representing a multi-channel audio signal using a downmix signal. In many embodiments, the method may allow reduced complexity and / or resource usage at the decoder / generation / reconstruction side at the expense of potentially increased complexity and / or resource usage at the encoder / source side.

[0018] The samples of the downmix audio signal may be time domain samples, or may be frequency domain samples (specifically subband samples). The samples of the auxiliary audio signal may be time domain samples, or may be frequency domain samples (specifically subband samples). The samples may span a specific time and frequency range.

[0019] The upmix parameterized data may include parameters (values) that relate the properties of the downmix signal to the properties of the multi-channel audio signal. The upmix parameterized data may include data indicating relative properties between channels of the multi-channel audio signal. The upmix parameterized data may include data indicating property differences between channels of the multi-channel audio signal. The upmix parameterized data may include data that is perceptually related to the synthesis of the multi-channel audio signal. The property may be, for example, a difference in phase and / or intensity and / or timing and / or correlation. In some embodiments and scenarios, the upmix parameterized data may represent abstract properties that cannot be directly understood by humans / experts (but may generally facilitate better reconstruction / lower data rates, etc.). The upmix parameterized data may include data including at least one of inter-channel intensity differences, inter-channel timing differences, inter-channel correlations, and / or inter-channel phase differences for channels of the multi-channel audio signal.

[0020] An ANN is a trained artificial neural network.

[0021] The artificial neural network may be a trained artificial neural network trained by training data, the training data comprising: a training downmix audio signal generated from a training multi-channel audio signal and training upmix parameterization data; the training employing a cost function for comparing the training multi-channel audio signal with an upmix multi-channel signal, the upmix multi-channel signal being generated from the training downmix signal and the generated auxiliary audio signal using the training upmix parameterization data. The artificial neural network may be a trained artificial neural network trained by training data, the training data comprising training data representing a series of related audio sources, the series of related audio sources comprising recordings of videos, movies, telecommunications, etc.

[0022] The artificial neural network may be a trained artificial neural network trained by training data having training input data comprising a training downmix audio signal of a training multi-channel audio signal and using a cost function, the cost function comprising a contribution indicative of a difference between: a training auxiliary audio signal generated by the second artificial neural network in response to the training data, and a training residual signal for the training downmix audio signal.

[0023] The generator may be arranged to generate the multi-channel audio signal by applying a matrix multiplication to the downmix signal and the auxiliary audio signal, wherein the coefficients of the matrix are determined from parameters of the upmix parameterization data. The matrix is ​​time and frequency dependent.

[0024] The audio device may specifically be an audio decoder device.

[0025] Control data values ​​may be referred to as metadata, conditioning characteristics, latent representations, conditioning variables, and / or composite terms.

[0026] According to an optional feature of the invention, the apparatus comprises a filter bank (401) arranged to generate a frequency subband representation of the downmix audio signal; and wherein the samples of the downmix audio signal are subband samples of the frequency subband representation.

[0027] Sub-band processing may provide particularly advantageous operation in many embodiments.The arrangement may be particularly suitable for sub-band processing, which may allow reduced complexity and / or generation of an improved multi-channel audio signal.

[0028] According to an optional feature of the invention, the artificial neural network (107) is an artificial neural network of a first plurality of subband artificial neural networks, each subband artificial neural network of the first plurality of subband neural networks being arranged to generate subband samples for a subband subset of a frequency subband representation of the auxiliary audio signal.

[0029] A particular advantage of this approach is that it may allow efficient sub-band processing, thereby allowing the required processing to be divided into multiple smaller artificial neural networks. This may generally allow reduced complexity and / or the generation of improved multi-channel audio signals.

[0030] In some embodiments, the first plurality of sub-band artificial neural networks comprises an artificial neural network for each sub-band of the frequency sub-band representation of the auxiliary audio signal.

[0031] In some embodiments, a number of input nodes for a neural network in the first plurality of sub-band neural networks is monotonically decreasing with increasing frequency.

[0032] In some embodiments, the generator is arranged to generate a frequency / subband representation of the multi-channel signal by performing upmixing of the frequency / subband representation of the auxiliary audio signal and the frequency / subband representation of the downmix audio signal, and convert the multi-channel signal frequency / subband representation of the multi-channel signal into a time domain representation of the multi-channel signal.

[0033] In many embodiments, each (or at least some) sub-band neural network is arranged to generate sub-band samples for one sub-band of the frequency sub-band representation of the auxiliary audio signal.

[0034] According to an optional feature of the invention, at least one control data value of the set of control data values ​​is processed by at least two artificial neural networks of the first plurality of sub-band artificial neural networks.

[0035] This may provide highly advantageous and efficient implementation and / or operation and / or performance in many embodiments and scenarios.

[0036] In some embodiments, at least some of said set of control data values ​​are common to a plurality of sub-bands of said frequency sub-band representation of said downmix audio signal.

[0037] According to an optional feature of the invention, at least one of the control data values ​​is not processed by at least one artificial neural network of the first plurality of sub-band artificial neural networks.

[0038] This may provide highly advantageous and efficient implementation and / or operation and / or performance in many embodiments and scenarios.

[0039] According to an optional feature of the invention, the artificial neural network 107 is trained by training data, the training data comprising at least one of the following: a training multi-channel audio signal, and a training downmix audio signal, and training upmix parameterization data generated from the training multi-channel audio signal; the training employs a cost function for comparing the training multi-channel audio signal with an upmix multi-channel signal, the upmix multi-channel signal being generated from the training downmix signal and from the generated auxiliary audio signal using the training upmix parameterization data.

[0040] This may provide a particularly efficient implementation and / or improved performance. In many embodiments, this may provide a particularly efficient and high performance training.

[0041] The set of control data values ​​generated by the subband neural network and common to multiple subbands can be input to multiple subband artificial neural networks for generating a multi-channel audio signal (and specifically, the second artificial neural network can be an artificial neural network among the multiple artificial neural networks that has a node for receiving the common set of control data values).

[0042] The artificial neural network may be trained by training data comprising a downmix signal and upmix parameterized data generated from a multi-channel audio signal and control data values; the training employing a cost function which compares the training multi-channel audio signal with an upmix multi-channel signal obtained by upmixing the training downmix signal and an auxiliary signal at an output of the artificial neural network using the training upmix parameterized data.

[0043] According to an optional feature of the invention, the artificial neural network (107) is trained by training data having training input data comprising a training downmix audio signal of a training multi-channel audio signal and a training set of control data values ​​for the training downmix audio signal and using a cost function, the cost function comprising a contribution indicative of a difference between a training auxiliary audio signal generated by the artificial neural network in response to the training data and a training residual signal for the training downmix audio signal.

[0044] This may provide a particularly efficient implementation and / or improved performance. In many embodiments, this may provide a particularly efficient and high performance training.

[0045] According to an optional feature of the invention, the cost function further comprises a contribution indicating a correlation between the auxiliary audio signal and the downmix of the multi-channel signal.

[0046] This may provide a particularly efficient implementation and / or improved performance. In many embodiments, this may provide a particularly efficient and high performance training.

[0047] According to an optional feature of the invention, the apparatus further comprises an upsampler for temporally upsampling the set of control data values, and wherein the artificial neural network (107) is arranged to include contributions from the upsampled control data values.

[0048] This may provide a particularly efficient implementation and / or improved performance.

[0049] According to an aspect of the present invention, there is provided an apparatus for generating a data signal representing a multi-channel audio signal, the apparatus comprising: a downmixer arranged to generate a downmix audio signal by downmixing the multi-channel audio signal, the downmixer further arranged to generate upmix parameterization data for upmixing the downmix audio; an artificial neural network arranged to generate a set of control data values ​​for adapting processing to generate an auxiliary audio signal for upmixing from the downmix audio signal, the artificial neural network having an output node for providing the set of control data values ​​and an input node for receiving samples of at least one of the downmix audio signal and the multi-channel audio signal; a generator arranged to generate the data signal to include the downmix audio signal, the upmix parameterization data and the set of control data values.

[0050] In many embodiments, the method may provide an improved audio experience. For many signals and scenarios, the method may provide an improved generation / reconstruction of a multi-channel audio signal with improved perceived audio quality. The method may provide a particularly advantageous arrangement, which in many embodiments and scenarios may allow for the possibility of facilitation and / or improvement of utilizing artificial neural networks in audio processing (typically including audio encoding and / or decoding). The method may allow for the advantageous employment of artificial neural networks when generating a multi-channel audio signal from a downmix audio signal.

[0051] The method may provide an efficient implementation and in many embodiments may allow reduced complexity and / or resource usage. The method may in many cases allow a data rate reduction for data representing a multi-channel audio signal using a downmix signal. In many embodiments, the method may allow reduced complexity and / or resource usage at the decoder / generation / reconstruction side at the expense of potentially increased complexity and / or resource usage at the encoder / source side.

[0052] The comments previously provided with respect to the apparatus for generating a multi-channel audio signal can be applied mutatis mutandis to the apparatus for generating a data signal.

[0053] The samples of the downmix audio signal may be time domain samples, or may be frequency domain samples (specifically subband domain samples). The samples of the auxiliary audio signal may be time domain samples, or may be frequency domain samples (specifically subband domain samples).

[0054] An ANN is a trained artificial neural network.

[0055] The artificial neural network may be a trained artificial neural network trained by training data, the training data comprising: a training downmix audio signal generated from a training multi-channel audio signal and training upmix parameterization data; the training employing a cost function for comparing the training multi-channel audio signal with an upmix multi-channel signal, the upmix multi-channel signal being generated from the training downmix signal and the generated auxiliary audio signal using the training upmix parameterization data.

[0056] The artificial neural network may be a trained artificial neural network trained by training data having training input data comprising a training downmix audio signal of a training multi-channel audio signal and using a cost function, the cost function comprising a contribution indicative of a difference between: a training auxiliary audio signal generated by the second artificial neural network in response to the training data, and a training residual signal for the training downmix audio signal.

[0057] The audio device may specifically be an audio encoder device.

[0058] Control data values ​​may be referred to as metadata, conditioning characteristics, latent representations, conditioning variables, and / or composite terms.

[0059] According to an optional feature of the invention, the set of control data values ​​provides a potential representation of at least one of the downmix audio signal and the multi-channel audio signal.

[0060] This may provide a particularly efficient implementation and / or improved performance.

[0061] According to an optional feature of the invention, the artificial neural network is trained by training data having training input data being samples of a training audio signal, the training audio signal being at least one of a multi-channel audio signal and a downmix of a multi-channel audio signal, and using a cost function, the cost function comprising a component for indicating a difference between the training audio signal and an output audio signal generated in accordance with a set of control data values ​​generated by the artificial neural network for the training audio signal.

[0062] This may provide a particularly efficient implementation and / or improved performance. In many embodiments, this may provide a particularly efficient and high performance training.

[0063] According to an optional feature of the invention, the output audio signal is an output of a synthetic artificial neural network having an input node for receiving a set of control data values ​​generated by the artificial neural network for the training audio signal, and the artificial neural network and the synthetic artificial neural network are jointly trained.

[0064] This may provide a particularly efficient implementation and / or improved performance. In many embodiments, this may provide a particularly efficient and high performance training.

[0065] According to an optional feature of the invention, the training audio signal and the output audio signal are downmixed audio channels.

[0066] According to an aspect of the present invention, a method for generating a multi-channel audio signal is provided, the method comprising: receiving a data signal, the data signal comprising: a downmix audio signal for the multi-channel signal; upmix parameterization data for upmixing the downmix audio signal; a set of control data values ​​for adapting processing to generate an auxiliary signal for upmixing from the downmix audio signal; an artificial neural network (107) having an input node for receiving samples of the downmix audio signal and an output node for providing samples of the auxiliary audio signal, the artificial neural network (107) further comprising a node arranged to include a contribution from the set of control data values; and generating the multi-channel audio signal from the downmix signal and the auxiliary signal according to the upmix parameterization data.

[0067] According to an aspect of the invention, there is provided a method for generating a data signal representing a multi-channel audio signal, the method comprising: generating a downmix audio signal by downmixing the multi-channel audio signal, the downmixer being further arranged to generate upmix parameterization data for upmixing the downmix audio; an artificial neural network generating a set of control data values ​​for adapting processing to generate an auxiliary audio signal for upmixing from the downmix audio signal, the artificial neural network having an output node for providing the set of control data values ​​and an input node for receiving samples of at least one of the downmix audio signal and the multi-channel audio signal; generating the data signal to include the downmix audio signal, the upmix parameterization data and the set of control data values.

[0068] These and other aspects, features and advantages of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings, in which

[0070] Figure 1shows some elements of an example of an audio device according to some embodiments of the present invention;

[0071] Figure 2 An example of the structure of an artificial neural network is shown;

[0072] Figure 3 An example of a node of an artificial neural network is shown;

[0073] Figure 4 shows some elements of an example of an audio device according to some embodiments of the present invention;

[0074] Figure 5 shows some elements of an example of an audio device according to some embodiments of the present invention;

[0075] Figure 6 shows some elements of an example of an audio device according to some embodiments of the present invention;

[0076] Figure 7 shows some elements of an example of an apparatus for training an artificial neural network for an audio device according to some embodiments of the present invention; and

[0077] Figure 8 Some elements of a possible arrangement of a processor for implementing elements of an audio device according to some embodiments of the invention are shown. DETAILED DESCRIPTION

[0078] Figure 1 Some elements of an audio device according to some embodiments of the invention are shown.

[0079] The audio device comprises a receiver 101 arranged to receive a data signal / bitstream comprising a downmix audio signal which is a downmix of a multi-channel audio signal. The following description will focus on the case where the multi-channel audio signal is a stereo signal and the downmix signal is a mono signal, but it will be appreciated that the methods and principles described are equally applicable to multi-channel audio signals having more than two channels and to downmix signals having more than a single channel (although fewer channels than a multi-channel audio signal).

[0080] In addition, the received data signal includes upmix parameterized data for upmixing the downmix audio signal. The upmix parameterized data may specifically be a set of parameters indicating the relationship between signals of different audio channels of a multi-channel audio signal (specifically, a stereo signal) and / or between the downmix signal and the audio channels of the multi-channel audio signal. Typically, the upmix parameters may indicate a time difference, a phase difference, a level / intensity difference and / or a similarity measure, such as correlation. Typically, upmix parameters are provided on a time-by-time and frequency-by-frequency basis (time-frequency slices). For example, new parameters may be provided periodically for a subband set. Parameters may specifically include inter-channel phase difference (IPD), overall phase expansion (OPD), inter-channel correlation (ICC), channel phase difference (CPD) parameters as known from parametric stereo coding (and from higher channel coding).

[0081] Typically, the downmix audio signal is encoded, and in a specific example, the receiver 101 includes a decoder 103 that decodes the downmix audio signal (i.e., the mono signal). It should be understood that in the case where the received downmix audio signal is not encoded, the decoder 103 may not be needed and the decoder 103 may be considered as an integral part of the receiver 101.

[0082] The receiver 101 is coupled to a generator 105, which generates a multi-channel audio signal from the downmix signal. The generator 105 is arranged to generate the multi-channel audio signal from the downmix audio signal and from the auxiliary audio signal according to the parameterized upmix data. The generator can generate the output multi-channel audio signal specifically for the stereo case by applying a time- and frequency-dependent 2x2 matrix multiplication to the samples of the downmix audio signal and the auxiliary audio signal. The coefficients of the 2x2 matrix are usually determined from the upmix parameters of the upmix parameterized data on the basis of time and frequency band. For other upmix operations, such as from a mono or stereo downmix signal to a five-channel multichannel audio signal, the generator 105 can apply a matrix multiplication of a matrix with a suitable size.

[0083] It should be understood that many different methods for generating such a multi-channel audio signal from the downmix audio signal and the auxiliary audio signal, and for determining suitable matrix coefficients from the upmix parameterized data are known to those skilled in the art, and any suitable method may be used. In particular, various methods for parameterized stereo upmixing based on downmix and auxiliary audio signals are well known to those skilled in the art.

[0084] In conventional systems, upmixing includes generating an auxiliary audio signal in the form of a decorrelated signal of a monophonic audio signal. It has been found that by generating a decorrelated signal and mixing it with the monophonic audio signal, the quality of the upmixed signal is improved, and decoders have therefore been developed to take advantage of this. The decorrelated signal is typically generated by a decorrelator in the form of an all-phase filter applied to the monophonic audio signal. However, although the use of such an all-pass filter often results in the generation of a multi-channel audio signal that is perceived as having improved quality, it is still not ideal and some audio quality degradation is typically perceptible.

[0085] Figure 1 The audio device uses methods that have been found to tend to provide improved perceived audio quality in many scenarios and for many different audio signals.

[0086] In this method, instead of generating a decorrelated signal by direct filtering of the downmix / mono audio signal, an auxiliary audio signal is generated by a trained artificial neural network, where the auxiliary audio signal is used by the generator 105 to generate a multichannel audio signal based on the upmix parameters.

[0087] The apparatus further comprises an artificial neural network, which will also be referred to as synthesis artificial neural network 107. The synthesis artificial neural network 107 is coupled to the decoder 103 / receiver 105. The synthesis artificial neural network 107 comprises in particular an input node for receiving samples of the downmix audio signal. The synthesis artificial neural network 107 has an output node for providing samples of the auxiliary audio signal, and these samples can then be fed to the generator 105 where the upmix operation is done.

[0088] In the method, the upmixing process is therefore not based on applying a decorrelation filter to the downmix audio signal to generate a decorrelated signal, which is then combined with the downmix audio signal to generate a multi-channel audio signal. Instead, the trained artificial neural network generates an auxiliary audio signal, which can specifically replace the decorrelated signal used in a conventional upmix decoder.

[0089] As will be described in detail later, different methods can be used to train the artificial neural network, and in particular, the overall training can seek to make the output of the audio device a multi-channel audio signal that corresponds most closely to the original multi-channel audio signal. Therefore, the synthetic artificial neural network 107 can be trained to provide an auxiliary audio signal that most effectively leads to accurate reconstruction of the multi-channel audio signal. Compared with conventional methods, such signals are not necessarily decorrelated signals, but the synthetic artificial neural network 107 will be trained to cause an auxiliary audio signal that is most suitable for combining with the downmix audio signal to generate a multi-channel audio signal. This signal is usually not a decorrelated signal of the downmix audio signal, but can be, for example, a partially correlated signal, and in fact in many cases may be closer to the actual residual signal caused by the original downmix of the multi-channel audio signal. Therefore, users of the trained artificial neural network can allow the decoder to inherently and automatically consider and compensate for the effects that may be introduced at the encoder side.

[0090] In the method, the artificial neural network arrangement is accordingly arranged to generate a second auxiliary "helper" signal, which assists and improves the multi-channel reconstruction. For example, for a stereo signal, the encoder can generate a downmix signal as c*(l+r), where l and r represent the left channel signal and the right channel signal respectively, and c is a time and frequency related scaling factor. The corresponding second signal for the ideal reconstruction can be shown as d*(lr), where d is again time and frequency related. The two signals are not necessarily completely decorrelated, and a substantial advantage of the described method is that the artificial neural network arrangement will generate an auxiliary audio signal that will tend to approach the ideal signal d*(lr) compared to merely seeking to attempt decorrelation with a single downmix. This can generally provide a substantially improved reconstruction of the original multi-channel audio signal.

[0091] The generation of the auxiliary audio signal is further improved by a synthesis artificial neural network 107, which adapts the signal based on a set of control data values ​​received together with the downmix audio signal. In the method, the synthesis artificial neural network 107 also includes nodes for receiving contributions from control values ​​in the set of control data values ​​received by the receiver 101, and the contributions are usually provided to the synthesis device from the source device. Such nodes can be input nodes of an input layer, which also includes nodes for receiving samples of the downmix audio signal, or one, more or all of the nodes receiving control data values ​​can be nodes of a layer different from the input layer of the downmix audio signal. For example, some or all of the nodes receiving contributions from the control data values ​​can be part of a hidden layer or processing layer of the synthesis artificial neural network 107.

[0092] In the method, a specific arrangement is used accordingly so as to achieve that the processing and operation of the synthesis device can be modified / adapted remotely, and in particular, a method is used that allows encoder-side control of the decoder-side reconstruction of the multi-channel audio signal. The method specifically provides a way for the encoder-side to control the generation of the auxiliary audio signal and for adapting the operation of the trained local artificial neural network, which performs the components of the synthesis of the multi-channel audio signal. The method can, for example, allow the encoder device to generate signal-specific / dependent control data, which can adapt the operation of the decoder to generate an improved auxiliary audio signal. In addition, the method specifically allows remote control of functions that may be unknown or unpredictable at the encoder / source side, or in fact even at the decoder / synthesis side. The method can specifically provide encoder / remote control of the artificial neural network so that this can be adapted to provide more desired or improved operation. In addition, in some embodiments, the method can allow the complexity of the synthesis device to be reduced, because for example, signal-specific control data can be generated at the source side, rather than requiring the synthesis device to process the signal to generate data that can adapt the second artificial neural network 109.

[0093] In many embodiments, the method allows for improved reconstruction of multi-channel audio signals and may provide an efficient implementation that may be realized with relatively low complexity and resource usage.

[0094] An artificial neural network as used in the described functionality may be a network of nodes arranged in layers and each node holding a node value. Figure 2 An example of a portion of an artificial neural network is shown.

[0095] The node value for a given node can be calculated as a contribution including some or often all nodes from the previous layer of the artificial neural network. Specifically, the node value for a node can be calculated as a weighted sum of the node values ​​output by all nodes of the previous layer. Typically, a bias can be added, and the result can be affected by an activation function. The activation function provides the necessary parts of each neuron by typically providing nonlinearity. This nonlinearity and activation function provide a significant impact in the learning and adaptation process of the neural network. Therefore, the node value is generated based on the node value of the previous layer.

[0096] The artificial neural network may specifically include an input layer 201, which includes a plurality of nodes receiving input data values ​​for the artificial neural network. Therefore, the node values ​​of the nodes of the input layer may generally be directly the input data values ​​for the artificial neural network, and therefore may not be calculated based on other node values.

[0097] The artificial neural network may further include, not include, one or more hidden layers 203 or processing layers. For each of such layers, the node value is typically generated based on the node values ​​of the nodes of the previous layer, and specifically, the weighted combination and added bias are followed by an activation function (such as sigmoid, ReLU or Tanh function can be applied).

[0098] Specifically, Figure 3 As shown, each node (which may also be called a neuron) can receive input values ​​(from nodes in the previous layer) and thereby compute a node value as a function of these values. Typically, this involves first generating a value that is a linear combination of the input values, each of which is weighted by a weight:

[0099]

[0100] Where w refers to the weight, x refers to the node of the previous layer, and n refers to the index of the different nodes of the previous layer.

[0101] An activation function can then be applied to the resulting combination. For example, the node value l can be determined as:

[0102] l=f(k)

[0103] The function may be, for example, a rectified linear unit function as described in Xavier Glorot, Antoine Bordes, Yoshua Bengio, Proceedings of the Fourtec International Conference on Artificial Intelligence and Statistics PML 15: 315-323, 2011:

[0104] f(k)=ReLU(k)=max(0,k)

[0105] Other commonly used functions include sigmoid function or tanh function. In many embodiments, multiple functions can be used to calculate node output or value. For example, both ReLU and Sigmoid functions can be combined using activation functions such as:

[0106] f(k)=ReLU(k)+σ(k)

[0107] Such operations can be performed by every node of the artificial neural network (usually except the input nodes).

[0108] The artificial neural network also includes an output layer 205, which provides the output from the artificial neural network, i.e. the output data of the artificial neural network are the node values ​​of the output layer. For the hidden / processing layers, the output node values ​​are generated by a function of the node values ​​of the previous layers. However, in contrast to the hidden / processing layers, where the node values ​​are generally not accessible or further used, the node values ​​of the output layer are accessible and provide the result of the operation of the artificial neural network.

[0109] Several different network structures and toolboxes have been developed for artificial neural networks, and in many embodiments, artificial neural networks can be based on adapting and customizing such networks. An example of a network architecture that may be suitable for the above applications is the WaveNet of van den Oord et al., which is described in Oord, Aaron van den, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. "Wavenet: A generative model for raw audio." arXiv preprint arXiv:1609.03499 (2016).

[0110] WaveNet is an architecture for synthesizing time-domain signals using dilated causal convolutions and has been successfully applied to audio signals. For WaveNet, the following activation functions are usually used:

[0111] z=tanh(W f，k *x)⊙σ(W g，k *x) where * represents the convolution operator, ⊙ represents the element-wise multiplication operator, σ(·) is the sigmoid function, k is the layer index, f and g represent the filter and gate, respectively, and W represents the weights of the learned artificial neural network. The filter product of the equation can generally provide a filtering effect, where the gated product provides a weighting of the result, which in many cases can effectively allow the contribution of a node to be reduced to essentially zero (i.e., it can allow or "cut off" a node that provides contributions to other nodes, thereby providing a "gate" function). In different cases, the gate function can cause the output of the node to be negligible, while in other cases it will substantially contribute to the output. Such a function can substantially help allow the neural network to learn and train efficiently.

[0112] In some cases, the artificial neural network can be further arranged to include additional contributions that allow the artificial neural network to be dynamically adapted or customized for specific desired properties or characteristics of the generated output. For example, a set of values ​​can be provided to adapt the artificial neural network. For example, a set of values ​​can be provided to adapt the artificial neural network. These values ​​can be included by providing contributions to some nodes of the artificial neural network. These nodes can be specific input nodes, but can generally be nodes of hidden layers or processing layers. Such adaptation values ​​can, for example, be weighted and added as a contribution to the weighted sum / correlation value for a given node. For example, for WaveNet, such adaptation values ​​can be included in the activation function. For example, the output of the activation function can be given as:

[0113] z=tanh(W f，k *x+V f，k *y)⊙σ(W g，k *x+V g，k *y)

[0114] where y is a vector representing the adaptation values, and V represents the appropriate weights for these values.

[0115] The above description relates to a neural network approach that may be applicable to many embodiments and implementations. However, it should be understood that many other types and structures of neural networks may be used. Indeed, many different methods for generating neural networks have been and are being developed, including neural networks that use complex structures and processes different from those described above. The method is not limited to any specific neural network method, and any suitable method may be used without detracting from the invention.

[0116] In many embodiments, the set of control data values ​​is generated by an audio device, which will also be referred to as an encoder or source audio device. The source audio device may provide a data signal, and in particular a bitstream, the bitstream comprising a downmix audio signal, upmix parameterization data and a set of control data values. The source audio device may in particular be an encoder device receiving a multi-channel audio signal, the encoder device encoding the multi-channel audio signal by generating a downmix audio signal with associated upmix parameterization data. In particular, the source audio device may be a parametric stereo (PS) encoder, which receives a stereo signal and encodes it into a mono audio signal with associated upmix parameterization data.

[0117] In addition, the source audio device is arranged to include a set of control data values ​​in the bitstream to allow the synthesized audio device to adapt its upmixing process. In particular, the source audio device may generate and send a set of control data values ​​that allows it to adapt the upmixing process of the synthesized audio device, and in particular it may adapt the operation of the artificial neural network to generate the auxiliary audio signal at the synthesized audio device.

[0118] The source audio device may use different methods to generate a set of control data values. Indeed, in some embodiments, the source audio device may store a predetermined set of control data values ​​that may be adapted to the upmixing operation of the synthetic audio device, and in particular, the artificial neural network operation. This may, for example, allow different source audio devices to adapt the operation of the synthetic audio device to their individual preferences. As another example, the method may allow adaptation of the audio system without the need to update all end-user devices. For example, if improved training or adaptation to different types of signals is introduced into the system, this may be achieved by modifying the set of control data values ​​stored in the source audio device. This may allow updating of the synthetic artificial neural network to perform improved operation without the need to retrain or modify the synthetic artificial neural network itself.

[0119] In many embodiments, the source audio device may be arranged to determine a set of control data values ​​based on the multi-channel audio signal or the generated downmix audio signal.

[0120] In some embodiments, one or more of the control data values ​​may be generated by analysis of the downmix audio signal.

[0121] It will be appreciated that many different algorithms and processes are known for analysing audio signals to extract or determine properties of the audio signals, and that any suitable (analysis) methods, algorithms and properties may be used.

[0122] In many embodiments, the source audio device may comprise functionality for generating a set of control data values ​​based on an artificial neural network, the artificial neural network generating the set of control data values ​​from the downmix audio signal. Figure 4 An example of such an encoder is shown in .

[0123] Figure 4 The source audio device of 400 comprises a downmixer 401, which receives a multi-channel audio signal, which in the present case is a stereo signal. The downmixer 401 generates a downmix audio signal, which in a particular instance is a mono signal. In addition, it generates upmix parameters, such as IPD, OPD, ICC, CPD parameters. It can further encode the downmix audio signal in a suitable format. It should be understood that many different algorithms and methods for generating a downmix audio signal with associated upmix parameters are known, and any suitable method can be used (e.g., in the present case, PS encoding can be applied).

[0124] The (encoded) downmix audio signal and the upmix parameters are fed to a data signal generator 403 which is arranged to generate an output data signal / bitstream comprising the (encoded) downmix audio signal and upmix parameterization data comprising the upmix parameters.

[0125] The source audio device further comprises an artificial neural network which will be referred to hereinafter as source artificial neural network 405. The source artificial neural network 405 receives samples of the downmix audio signal and / or the multi-channel audio signal and in response it generates a set of control data values. Such a set of control data values ​​is fed to the data signal generator 403, where they are included in the generated bitstream.

[0126] Therefore, the source audio device specifically comprises a source artificial neural network 405, which is arranged to receive samples of the downmix audio signal and / or the multi-channel audio signal, wherein these samples are fed to an input node of the source artificial neural network 405. The output node of the source artificial neural network 405 provides a set of control data values ​​for the downmix audio signal / multi-channel audio signal. The set of control data values ​​may comprise a number of values ​​(e.g. specific scalar values) which reflect properties or characteristics of the downmix audio signal / multi-channel audio signal.

[0127] Typically, the source artificial neural network 405 may include a much larger number of input nodes than output nodes, and thus generate a relatively low number of control data values ​​based on a relatively large number of samples. Therefore, the control data values ​​can provide a highly compressed and simplified representation of some properties or characteristics of the downmix audio signal / multi-channel audio signal. For example, the source artificial neural network 405 can have 1028 input nodes and 16 output nodes, thereby providing a highly compressed set of values ​​depending on the properties of the downmix audio signal. Potential representations of the downmix audio signal and / or the multi-channel audio signal can be generated.

[0128] In this arrangement, the output of the source artificial neural network 405 is introduced (via data signal distribution / communication) into the synthetic artificial neural network 107 and is therefore used to control and adapt the processing of the synthetic artificial neural network 107. Therefore, even with constant weights and coefficients, the synthetic artificial neural network 107 is not just a fixed filter or trained network, but can be regarded as an adaptive or variable operation that adapts based on the results of the source artificial neural network 405.

[0129] Similarly, the source artificial neural network 405 and the generated control data values ​​are not specifically control data values ​​for representing a particular selected property or characteristic of a signal. Instead, the source artificial neural network 405 can be trained so that it automatically adapts to provide control data values ​​that are particularly suitable for adapting the synthetic artificial neural network 107 to provide improved output values ​​for accurate reconstruction of the multi-channel audio signal.

[0130] This method may provide a particularly efficient method, which may generally lead to an improved reconstruction of the multi-channel audio signal at the synthesis side.Thus, the method may generate a downmix audio signal and / or a latent representation of the multi-channel audio signal.

[0131] In many embodiments, the synthesized audio device is arranged to perform sub-band processing. Figure 5 As shown, Figure 1 The device may be modified to include a filter bank arranged to generate a frequency subband representation of the downmix audio signal. The filter bank may be a quadrature mirror filter (QMF) bank, or may be implemented, for example, by a fast Fourier transform (FFT), but it will be appreciated that many other filter banks and methods for dividing an audio signal into a plurality of subband signals are known and may be used. The filter bank may specifically be a complex-valued pseudo-QMF bank, which generates, for example, 32 or 64 complex-valued subband signals.

[0132] In many embodiments, filter bank 501 is arranged to generate a subband signal set for subbands with equal bandwidth. In other embodiments, filter bank 501 can be arranged to generate subband signals with subbands with different bandwidths. For example, higher frequency subbands can have higher bandwidths than lower frequency subbands. In addition, subbands can be grouped together to form higher bandwidth subbands.

[0133] Typically, a sub-band may have a bandwidth in the range of 10 Hz to 10,000 Hz.

[0134] In some such embodiments, the artificial neural network generating samples for the auxiliary audio signal is arranged to receive sub-band samples, i.e. samples of a sub-band audio signal. Figure 5 The apparatus comprises a plurality of subband neural networks 109, each of which is arranged to receive subband samples for a subband generated by a filter bank from a downmix signal. Each of these subband artificial neural networks further receives a set of control data values, and it proceeds to generate subband samples of the multi-channel audio signal for that subband.

[0135] The subband samples from the subband artificial neural network are fed to the generator 105, which proceeds to generate the reconstructed multi-channel audio signal. For example, in some embodiments where a subband representation of the multi-channel audio signal is desired (e.g., because subsequent processing is also based on subbands), the generator 105 may simply output the subband samples from the subband artificial neural network according to a specific structure or format. In many embodiments, the generator 105 includes a function for converting the subband representation of the reconstructed multi-channel audio signal into a time domain representation. The generator 105 may specifically include a synthesis filter bank for performing an inverse operation of the filter bank 501, thereby converting the subband representation into a time domain representation of the multi-channel audio signal.

[0136] The generator may be specifically arranged to generate a frequency / subband domain representation of the multi-channel audio signal by processing a frequency or subband domain representation of the downmix audio signal and a frequency / subband domain representation of the auxiliary audio signal. Thus, the processing of the generator 105 may be a subband processing, such as, for example, a matrix multiplication performed on each subband on the subband samples of the downmix audio signal and the auxiliary audio signal generated by the corresponding subband artificial neural network.

[0137] The resulting subband / frequency domain representation may then be used directly, or may be converted to a time domain representation, for example using a suitable synthesis filter bank, which may in particular be applied by a separate synthesis filter for each channel.

[0138] Each of the plurality of sub-band neural networks may also be referred to as a synthetic sub-band artificial neural network. The comments previously provided with respect to the artificial neural network 107 will also apply mutatis mutandis to the synthetic sub-band artificial neural network, and in fact the synthetic artificial neural network 107 may be considered as one of the synthetic sub-band artificial neural networks.

[0139] Thus, in this arrangement each of the synthesis subband artificial neural networks receives subband samples for a subband of the synthesis subband artificial neural network and further all synthesis subband artificial neural networks are arranged to receive a set of control data values ​​from the source artificial neural network 405 .

[0140] Each of the synthesis subband artificial neural networks generates subband samples for a subband subset of a frequency subband representation of the auxiliary audio signal, and typically (only) for the subbands for which it receives input samples from the filterbank 501 .

[0141] In many embodiments, the apparatus comprises an artificial neural network for each subband of the frequency subband representation of the auxiliary audio signal generated by the filter bank. Thus, in many embodiments, the output samples for each subband of the filter bank 501 are fed to the input node of a synthesis subband artificial neural network, which then generates subband samples of the auxiliary audio signal for that subband. In many embodiments, the subband processing can therefore be completely separated for each subband.

[0142] In this example, the generation of the auxiliary audio signal is therefore performed on a subband basis using separate and individual artificial neural networks in each subband. The individual artificial neural networks are trained to provide output samples for the subbands for which the input subband samples are provided. However, the subband artificial neural networks are further adapted based on the control data values ​​received by the receiver 101.

[0143] It has been found that this method provides a very advantageous generation of auxiliary audio signals, which allows extremely high quality reconstruction of multi-channel audio signals. In addition, it can allow efficient operation with significantly reduced complexity and / or generally significantly reduced computing resource requirements. Subband artificial neural networks tend to be substantially smaller than a single complete artificial neural network required for generating the entire signal. Typically, far fewer nodes are required for processing, and possibly even fewer layers, resulting in a very large reduction in the number of operations and calculations required to implement the artificial neural network functions. Although more artificial neural networks are needed to cover all subbands, smaller artificial neural networks will generally result in a huge reduction in the total number of operations required, and therefore a huge reduction in overall computing resource requirements. In addition, in many scenarios, it can allow a more efficient learning process.

[0144] The subband arrangement can accordingly provide a computationally efficient method for allowing the implementation of an artificial neural network to assist in the decoding of audio data including a downmix audio signal and upmix parameterized data. The described system and method allow the reconstruction of a high-quality multi-channel audio signal, and a generally significantly improved audio quality can be achieved compared to conventional methods. In addition, a computationally efficient decoding process can be achieved. The subband and artificial neural network based methods can also be compatible with other processes that use subband processing.

[0145] In some embodiments, subband processing can be more flexible than strict subband by subband processing. For example, in some embodiments, each synthetic subband artificial neural network can receive subband samples not only from the subband itself, but also possibly for one or more other subbands. For example, in some embodiments, the synthetic subband artificial neural network for one subband can also receive samples of the downmixed audio signal from one or two neighboring / adjacent subbands. As another example, in some embodiments, one or more of the synthetic subband artificial neural networks can also receive input samples from one or more subbands including harmonics (or sub-harmonics) of the frequency for the subband. For example, a subband around a 500 Hz center frequency can also receive frequencies from a subband around a 1000 Hz center frequency. Such additional subbands having a specific relationship to the subbands of the synthetic subband artificial neural network can provide additional information, which can allow the generation of improved synthetic subband artificial neural networks for some audio signals.

[0146] In some embodiments, all synthetic sub-band artificial neural networks may have the same properties and dimensions. In particular, in many embodiments, all synthetic sub-band artificial neural networks may have the same number of input nodes and output nodes, and may have the same internal structure. This approach may be used, for example, in embodiments where all sub-bands have the same bandwidth.

[0147] However, in some embodiments, the synthesized sub-band artificial neural network may comprise non-identical neural networks. In particular, in some embodiments, the number of input nodes for the synthesized sub-band artificial neural network may be different for at least two of the artificial neural networks. Thus, in some embodiments, the number of input samples included in the determination of output samples may be different for different sub-bands and synthesized sub-band artificial neural networks.

[0148] In some embodiments, for some lower frequency sub-bands, the number of samples / input nodes may be higher than for some higher frequency bands. In fact, for increasing frequency, the number of samples / input nodes may decrease monotonically. The lower frequency synthesis sub-band artificial neural network may therefore be larger and take into account more input samples than the higher frequency synthesis sub-band artificial neural network. This approach may, for example, be combined with sub-bands having different bandwidths, such as when the lower frequency sub-band may have a higher bandwidth than the higher frequency bandwidth.

[0149] Such an approach may provide an improved trade-off between achievable audio quality and computational complexity and resource usage in many scenarios. It may provide a tighter adaptation of the system to reflect the typical characteristics of audio, allowing more efficient processing.

[0150] In the above method, a single source artificial neural network 405 is used to generate control data values ​​for a set of control data values, while a plurality of synthesis sub-band artificial neural networks processes the sub-band downmix audio signals.

[0151] In other embodiments, the generation of the set of control data values ​​in the source audio device may be based on sub-bands. Similarly, for a synthesized audio device, such as Figure 6 As shown, the source audio device may also include a filter bank 601, which is applied to the downmixed audio signal to provide a set of subband signals, which are then fed to corresponding subband artificial neural networks that each generate a set of control data values. Such multiple artificial neural networks will also be referred to as source subband artificial neural networks. The set of control data values ​​generated by the source subband artificial neural network can then be fed to the synthetic subband artificial neural network. The subbands generated by such a filter bank 601 do not need to be the same as the subbands generated by the filter bank 501 for generating subband samples for the synthetic subband artificial neural network. For example, each of the subbands used to determine the set of control data values ​​can include a different number of subbands for the synthetic subband artificial neural network, and the set of control data value sets determined by one source subband artificial neural network can be fed to the appropriate synthetic subband artificial neural network.

[0152] However, in many embodiments, the sub-band used to generate the control data values ​​by the source audio device may be the same sub-band as the sub-band used to power the synthesis sub-band artificial neural network of the receiver.

[0153] The comments previously provided with respect to the artificial neural network 405 will also apply mutatis mutandis to the source sub-band artificial neural network, and in fact the source artificial neural network 405 may be considered as one of the source sub-band artificial neural networks.

[0154] In the method, each source subband artificial neural network generates a set of control data values, which in some embodiments are only applied to a subset of the synthetic subband artificial neural networks. In fact, in many embodiments, each source subband artificial neural network can generate a set of control data values ​​for one of the synthetic subband artificial neural networks. In some embodiments, the device can include the same number of source subband artificial neural networks and synthetic subband artificial neural networks, and specifically for each subband of the filter bank 501, there can be one source subband artificial neural network and one synthetic subband artificial neural network. In other embodiments, there can be different numbers of source subband artificial neural networks and synthetic subband artificial neural networks. For example, a source subband artificial neural network can receive input samples for a group of subbands and generate a set of control data values ​​for the group of subbands. Then, the set of control data values ​​can be applied to the group of synthetic subband artificial neural networks for these subbands.

[0155] This subband-based approach for generating sets of control data values ​​can provide improved results in many scenarios by allowing more accurate sets of control data values ​​to be generated and used to adapt the resulting subband artificial neural network. It can also reduce complexity and / or resource usage in many scenarios. For example, fewer values ​​in the set of control data values ​​can be implemented in many embodiments, thereby allowing the complexity of both the source subband artificial neural network and the resulting subband artificial neural network to be reduced. In addition, typically, the source subband artificial neural network will have fewer inputs and be much smaller than an artificial neural network covering the entire bandwidth.

[0156] In many such embodiments, each source subband artificial neural network may generate a set of control data values ​​for a given subband subset and typically for a single subband based on the subband samples of the subband subset. However, in addition, one or more of the source subband artificial neural networks may also include subbands from one or more other subbands, i.e., an input to a source subband artificial neural network may have a receiving input node for subband samples for a subband for which the source subband artificial neural network does not generate any set of control data values.

[0157] As a specific example, in many embodiments, each source sub-band artificial neural network may receive as input not only sub-samples from the sub-band for which the set of control data values ​​is generated, but also sub-samples from neighboring sub-bands.

[0158] Such methods may generally allow an improved set of control data values ​​to be generated, which may result in improved audio quality. In particular, it has been found that taking surrounding subbands into account may allow the set of control data values ​​to better reflect the temporal resolution of the downmix audio signal / multi-channel audio signal. It has been found that such methods may in particular allow a better representation of temporal peaks.

[0159] In some embodiments, one or more of the source sub-band artificial neural networks, or in fact the source artificial neural network 405 (in the case where only one such artificial neural network is included), may further take as input sub-band samples outside the time interval of samples for which the corresponding synthesis sub-band artificial neural network generates the auxiliary audio signal.

[0160] In particular, the processing of the audio device operates on a frame-by-frame basis, wherein a time interval / frame of the received downmix audio signal is processed to generate output samples of the multi-channel audio signal for that time interval / frame. Thus, for each frame, the filter bank 501 generates sub-band samples, which are fed to a source sub-band artificial neural network for generating a set of control data values ​​for that time interval, and which are fed to a source sub-band artificial neural network for generating sub-band samples of a sub-band representation of the multi-channel audio signal based on the sub-bands and the set of control data values.

[0161] Hence, in particular, each source subband artificial neural network operates in blocks, wherein each operation in generating a set of output samples from a set of input samples corresponds to a time interval of the downmix audio signal / multi-channel audio signal for which an output sample of the multi-channel audio signal is generated.

[0162] In some embodiments, in addition to the appropriate subband samples generated for the current time interval, one or more of the source subband artificial neural networks may also receive subband samples for another time interval, typically from one or more adjacent time intervals. For example, in some embodiments, one or more of the source subband artificial neural networks may also include subband samples for the previous and next time intervals.

[0163] In many embodiments, such a method may provide for generating an improved set of control data values, resulting in improved audio quality.

[0164] In some embodiments, at least some of said set of control data values ​​are common to a plurality of sub-bands of said frequency sub-band representation of said downmix audio signal.

[0165] In the above method, the synthesis sub-band artificial neural networks are all provided with the same set of control data values. In some embodiments, the same set of control data values ​​may be provided to only some of the synthesis sub-band artificial neural networks. In particular, in some embodiments, at least one of the control data values ​​is processed by at least two synthesis sub-band artificial neural networks.

[0166] In many embodiments, using the same set of control data values ​​may provide improved efficiency and performance. It may generally reduce complexity and resource usage when generating the set of control data values. Furthermore, in many scenarios it may provide improved operation, and all available information provided by the control data values ​​may be considered by each synthesis sub-band artificial neural network, and thus improved adaptation of the synthesis sub-band artificial neural network may be achieved.

[0167] However, in some embodiments, different sets of control data values ​​may be provided to different synthesis sub-band artificial neural networks. In particular, in some embodiments, one or more control data values ​​will be processed by one of the synthesis sub-band artificial neural networks but not by another of the synthesis sub-band artificial neural networks.

[0168] For example, the source artificial neural network 405 may generate a set of control data values, and different subsets of these control data values ​​may be provided to different synthesis subband artificial neural networks. In other embodiments, some control data sets may also be generated to include some control data values, for example, provided manually or generated by analysis of the downmix audio signal. For example, harmonics or peaks may be detected in the downmix audio signal. Such data may, for example, be applied to only some of the synthesis subband artificial neural networks. For example, detected peaks or harmonics may be indicated only to the synthesis subband artificial neural networks in which the subbands in which they were detected.

[0169] In many embodiments, different sets of control data values ​​may be provided to different synthesis sub-band artificial neural networks. In many cases, this may include some control data values ​​being the same and some control data values ​​being different for different sets of synthesis sub-band artificial neural networks.

[0170] The above description focuses on applying sub-band processing to a downmix audio signal, and the source sub-band artificial neural network 107 generates a set of control data values ​​based on the downmix audio signal. However, it will be appreciated that the methods and principles described are equally applicable to multi-channel audio signals, and indeed, with minor modifications, references to the downmix audio signal may be considered equally to the multi-channel audio signal. Thus, in particular, sub-band processing may be applied to a multi-channel audio signal by applying a filter bank to the channels of the multi-channel audio signal.

[0171] Artificial neural networks are adapted to specific purposes through a training process, which is used to adapt / adjust / modify the weights and other parameters (e.g., bias) of the artificial neural network. It should be understood that many different training processes and algorithms are known for training artificial neural networks. Typically, training is based on a large training set, in which a large number of examples of input data are provided to the network. In addition, the output of the artificial neural network is typically (directly or indirectly) compared with an expected or ideal result. A cost function can be generated to reflect the expected results of the training process. In a typical scenario known as supervised learning, the cost function typically represents the distance between the prediction and the ground truth for specific input data. Based on the cost function, the weights can be changed, and by re-iterating the process for modifying the weights, the artificial neural network can be adapted to a state where the cost function is minimized.

[0172] In more detail, during the training step, the neural network can have two different information flows from input to output (forward pass) and from output to input (backward pass). In the forward pass, as described above, the data is processed by the neural network, and in the backward pass, the weights are updated to minimize the cost function. Typically, this backward propagation follows the gradient direction of the cost function pattern. In other words, by comparing the predicted output with the ground truth for a batch of data input, the direction in which the cost function is minimized can be estimated and propagated backward by updating the weights accordingly. Other methods for training artificial neural networks include, for example, the Levenberg-Marquardt algorithm, the conjugate gradient method, and Newton's method.

[0173] In the present case, the training may specifically include a training set including a possibly large number of multi-channel audio signals. In some embodiments, the training data may be a multi-channel audio signal in a time segment corresponding to a processing time interval of the trained artificial neural network, for example, the number of samples in the training multi-channel audio signal may correspond to the number of samples corresponding to the input nodes of the trained artificial neural network. Each training example may therefore correspond to an operation of the trained artificial neural network. However, typically, a batch of training samples is considered for each step to speed up the training process. In addition, many upgrades to the gradient descent are also possible to speed up convergence or avoid local minima in the cost function situation.

[0174] For each training multi-channel audio signal, the training processor may perform a downmix operation to generate a downmix audio signal and corresponding upmix parameterized data. Thus, the encoding process applied to the multi-channel audio signal during normal operation may also be applied to the training multi-channel audio signal, thereby generating downmix and upmix parameterized data.

[0175] Additionally, in some embodiments, the training processor may generate a residual signal reflecting the difference between the downmix audio signal and the multi-channel audio signal, or more generally representing parts of the multi-channel audio signal that are not properly represented by the downmix audio signal. For example, in many embodiments, the training processor may generate a downmix signal and may additionally generate a residual signal that, when upmixed based on the upmix parameterization data, will generate a (more) accurate multi-channel audio signal to be reconstructed.

[0176] Specifically, for stereo multi-channel audio signals, the training processor can use a parametric stereo scheme (e.g., according to a suitable standardization method). This encoding will apply a matrix operation related to frequency and time, such as a rotation operation, to the input stereo signal to generate a downmix signal and a residual signal. For example, a 2x2 matrix multiplication / complex value multiplication is usually applied to the input stereo signal, such as to substantially align one of the rotated channel signals to have a maximum signal value. The channel can be used as a mono signal, and the rotation is usually performed on the basis of a frame. The rotation value can be stored as part of the upmix parameterized data (or the parameters allowing determination thereof can be included in the upmix parameterized data). Therefore, in a synthesis device, an opposite rotation can be performed to reconstruct a stereo signal. The rotation of the stereo signal causes another stereo signal, in which one of the channels is therefore aligned with the maximum intensity. Another channel is usually discarded in a parametric stereo encoder to reduce the data rate. In conventional PS decoding, a decorrelated signal is usually generated at a decoder and used for an upmix process. In the current training method, this second signal may be used as a residual signal for the downmix, since it may represent the information discarded in the encoder and therefore it represents the ideal signal to be reconstructed in the decoder as part of the upmix process.

[0177] Thus, in some embodiments, the training processor may generate a training downmix signal and / or a training residual signal from the training multi-channel audio signal. The training downmix signal may be fed to an arrangement of a source artificial neural network 405 and a synthesis artificial neural network 107, or equivalently to an arrangement of a source subband artificial neural network and a synthesis subband artificial neural network, i.e. samples of the training downmix audio signal are fed to the neural network using the same processing (e.g. including subband filtering etc.) as is applied to the downmix audio signal by the audio device during normal operation.

[0178] Thus, in the method, a joint training of a source (subband) artificial neural network and a synthesis (subband) artificial neural network may be applied, wherein the training multi-channel audio signal generates input samples fed to the artificial neural networks in the source audio device and the synthesis audio device in the same way as they would be used for real signals when the source audio device and the synthesis audio device are in normal operation. The artificial neural networks of the source audio device and the synthesis audio device are accordingly trained together and the weights etc. of the two artificial neural networks are adapted together. In cases where only one device is available, a training setup may be used in which the missing artificial neural network is simulated / copied by a dedicated training artificial neural network.

[0179] For example, if the source audio device is trained independently of the synthetic audio device, the training setup may include the functionality of a standard or typical synthetic audio device (e.g., which may include an artificial neural network corresponding to the synthetic artificial neural network). In such cases, references to features of the synthetic audio device hereinafter may be considered to refer to corresponding features in the training setup.

[0180] Similarly, if the synthetic audio device is trained independently of any source audio device, the training setup may include the functionality of a standard or typical source audio device (e.g., it may include an artificial neural network corresponding to the source artificial neural network). In this case, references below to features of the source audio device may be considered to refer to corresponding features in the training setup.

[0181] Outputs from the neural network operation are then determined and a cost function is applied to determine a cost value for each training downmix audio signal and / or for a combined set of training downmix audio signals (eg, determine an average cost value for the training set). The cost function may include various components.

[0182] Typically, the cost function will include at least one component reflecting how close the generated signal is to a reference signal, the so-called reconstruction error. In some embodiments, the cost function will include at least one component reflecting how close the generated signal is to a reference signal from a perceptual viewpoint.

[0183] For example, in some embodiments, the auxiliary audio signal generated by the synthetic artificial neural network 107 for a given training downmix audio signal / multi-channel audio signal may be compared with a residual signal for the training downmix audio signal / multi-channel audio signal. A cost function contribution / combination reflecting the difference between the generated auxiliary audio signal and the reference residual signal may be generated. This process may be generated for all training sets to generate a total cost function.

[0184] Such an example is Figure 7 . In this example, the downmixer 701 may receive a training multi-channel audio signal, which in a specific example is a stereo signal. For a given training signal, the downmixer 701 performs a downmix operation to generate a training downmix signal, which in a specific example is a training mono audio signal. The downmix audio signal is fed to a preprocessor 703, which may specifically generate downmix audio signal samples for input to an artificial neural network. The preprocessor 703 specifically performs the same operations as performed in the source audio device and / or the synthesis device to generate samples for the artificial neural network, and in fact the same functions may generally be used, i.e., the functions for generating artificial neural network input samples are also used during the training process.

[0185] The output of the preprocessor 703 is Figure 1 The source artificial neural network 405 and the synthesis artificial neural network 107 are coupled to each other in the same manner as the audio device of the present invention. The source artificial neural network 405 and the synthesis artificial neural network 107 are coupled to each other so that the output set of control data values ​​generated by the source artificial neural network 405 is fed to the synthesis artificial neural network 107. Therefore, the output of the synthesis artificial neural network 107 corresponds to the samples of the auxiliary audio signal generated for the training downmix audio signal.

[0186] Therefore, for a given training multi-channel audio signal, Figure 7 The training system performs Figure 1 and Figure 4 The artificial neural network performs the same operation as the audio device, thereby generating an auxiliary audio signal that the audio device would generate if the artificial neural network had the same data / configuration (coefficients, biases, etc.).

[0187] The output of the synthetic artificial neural network 107 is fed to a comparator 705, which proceeds to compare the generated auxiliary audio signal with the residual signal generated by the downmixer 701. As mentioned, the residual signal in principle allows a substantially perfect reconstruction of the multi-channel audio signal and can therefore be regarded as a close approximation of the ideal auxiliary audio signal. The comparison between the generated auxiliary audio signal and the residual signal thus provides an indication of how favorable the generated auxiliary audio signal is. A cost value can therefore be determined accordingly based on the comparison, and in particular, the higher the difference, the higher the cost value.

[0188] It should be understood that many different methods can be used to determine the cost value reflecting the difference between the signals. For example, correlation can be performed using a cost value having a monotonically decreasing value for an increased correlation value. As another example, two signals can be subtracted from each other, and a power measurement for a difference signal can be used as a cost value. It should be understood that many other methods are available and can be used.

[0189] Thus, in this example, the cost function generates a cost value that reflects how closely the generated auxiliary audio signal matches the corresponding residual signal for the training multi-channel audio signal.

[0190] Based on the cost value, the training processor 707 can adapt the weights of the artificial neural network. For example, a back propagation method can be used. In particular, the training processor 707 can adjust the weights of both the source artificial neural network 405 and the synthetic artificial neural network 107 based on the cost value. For example, given the derivative of the weight with respect to the cost function (which represents the slope), the weight value is modified to move in the direction of the slope. For simple / minimal cases, in the case of back propagation with a single data input, reference can be made to the training of a perceptron (a single neuron).

[0191] This process may be iterated until the artificial neural network is considered trained. For example, a predetermined number of iterations of training may be performed. As another example, training may continue until the weights change by less than a predetermined amount. It is also very common to implement a validation stop where the network is again tested for a validation metric and stops when the expected results are achieved.

[0192] As a specific example, a stereo signal may be fed to a conventional PS downmix module that generates a downmix and an ideal residual signal, i.e. a residual signal that allows (close to) perfect reconstruction of the waveform at the decoder side. Using multiple pairs of mono signals and residual signals, the source artificial neural network 405 generates a set of control data values ​​that can provide a potential representation of the audio signal (frame). The set of control data values ​​is associated with the mono audio signal that is fed to the synthesis artificial neural network 107 for generating the auxiliary audio signal. When the artificial neural network is deep enough, such an arrangement can be trained using, for example, a RMSE (root mean square error of the auxiliary audio signal with respect to the residual signal) type cost function, so that the artificial neural network arrangement learns what the auxiliary audio signal should be for a given mono audio signal.

[0193] In this arrangement, the source artificial neural network 405 and the synthesis artificial neural network 107 are respectively jointly trained using the same downmix audio signal and cost function.The weights of both the source artificial neural network 405 and the synthesis artificial neural network 107 are updated based on the same training data and the same cost function and downmix audio signal.

[0194] In many embodiments, the residual signal can be used directly when compared to the generated auxiliary audio signal. However, more generally, a target audio signal generated from the residual audio signal can be generated and used for comparison with the generated auxiliary audio signal. The target audio signal can be generated by applying a function or signal processing application to the residual signal. For example, scaling / level setting of the residual audio signal can be applied to generate the target audio signal. As another example, a filter operation can be applied to the residual audio signal to generate the target audio signal. As another example, scaling can be applied to the residual signal to minimize the difference and / or maximize the correlation with the generated auxiliary audio signal.

[0195] In some embodiments, the cost function may be arranged alternatively or additionally to reflect the difference between the training multi-channel audio signal and the multi-channel audio signal generated by upmixing the downmix audio signal with the generated auxiliary audio signal. In such an example, the downmixer 701 may also generate upmix parameterized data that can be used in the upmix. Therefore, in some embodiments, the training may include reconstructing the multi-channel audio signal based on the generated auxiliary audio signal, rather than only training the artificial neural network to generate an auxiliary audio signal that matches the residual signal. The generation of the output multi-channel audio signal may be specifically performed using the same operation as that to be performed in the encoder. The output multi-channel audio signal may then be compared with the input multi-channel audio signal. Therefore, in some embodiments, the training data may alternatively or additionally include the multi-channel audio signal, rather than the training data substantially including the training downmix audio signal. The cost function may be based on a comparison of the generated auxiliary audio signal with the residual signal, or may be based, for example, on a comparison between the original training multi-channel audio signal and the reconstructed multi-channel audio signal.

[0196] In some embodiments, the cost function may also take into account other parameters. For example, in some embodiments, the cost function may further include taking into account the correlation between the generated auxiliary audio signal and the downmix signal. In particular, the more the downmix audio signal and the auxiliary audio signal are de-correlated, the cost function may indicate a lower cost. Increased de-correlation may indicate that there is less common information in both signals.

[0197] In an embodiment where subband operations are performed, the described method may be performed for each subband. Specifically, a residual audio signal may be generated for each subband, and the residual audio signal may be compared with the generated auxiliary audio signal subband samples for the subband. Thus, the cost function may be evaluated for each subband to determine a cost value, and the coefficients of the source subband artificial neural network and the synthesis subband artificial neural network for the subband may be trained.

[0198] In many embodiments, the source sub-band artificial neural network and / or indeed the source artificial neural network 405 may generate a set of control data values ​​that matches the time and frequency resolution of the synthesis sub-band artificial neural network, including in particular the synthesis artificial neural network 107 if only one artificial neural network is used. However, in other embodiments, the time and / or frequency resolution of the generated set of control data values ​​may be different, and in general the resolution may be lower for the set of control data values. In such cases, functionality may be included to modify the resolution. In particular, an interpolator may be incorporated to generate an interpolated set of parameter values ​​from the generated parameter values.

[0199] The audio device may be implemented in one or more appropriately programmed processors. In particular, the artificial neural network may be implemented in one or more such appropriately programmed processors. Different functional blocks, in particular the artificial neural network, may be implemented in separate processors and / or may be implemented, for example, in the same processor. Examples of suitable processors are provided below.

[0200] Figure 8 800 is a block diagram illustrating an example processor 800 according to an embodiment of the present disclosure. The processor 800 may be used to implement one or more processors that implement an apparatus or elements thereof as previously described (including, among other things, one or more artificial neural networks). The processor 800 may be any suitable processor type, including, but not limited to, a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), where the FPGA has been programmed to form a processor, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), where the ASIC has been designed to form a processor, or a combination thereof.

[0201] Processor 800 may include one or more cores 802. Core 802 may include one or more arithmetic logic units (ALUs) 804. In some embodiments, core 802 may include a floating point logic unit (FPLU) 806 and / or a digital signal processing unit (DSPU) 808 in addition to or in lieu of ALU 804.

[0202] The processor 800 may include one or more registers 812 communicatively coupled to the core 802. The registers 812 may be implemented using dedicated logic gates (e.g., flip-flops) and / or any memory technology. In some embodiments, the registers 812 may be implemented using static memory. The registers may provide data, instructions, and addresses to the core 802.

[0203] In some embodiments, the processor 800 may include one or more levels of cache memory 810 communicatively coupled to the core 802. The cache memory 810 may provide computer readable instructions to the core 802 for execution. The cache memory 810 may provide data for processing by the core 802. In some embodiments, the computer readable instructions may have been provided to the cache memory 810 by a local memory (e.g., a local memory attached to the external bus 816). The cache memory 810 may be implemented with any suitable cache memory type, for example, a metal oxide semiconductor (MOS) memory, such as a static random access memory (SRAM), a dynamic random access memory (DRAM), and / or any other suitable memory technology.

[0204] The processor 800 may include a controller 814 that may control inputs to the processor 800 from other processors and / or components included in the system and / or outputs from the processor 800 to other processors and / or components included in the system. The controller 814 may control data paths in the ALU 804, FPLU 806, and / or DSPU 808. The controller 814 may be implemented as one or more state machines, data paths, and / or dedicated control logic units. The gates of the controller 814 may be implemented as independent gates, FPGAs, ASICs, or any other suitable technology.

[0205] Registers 812 and cache 810 may communicate with controller 814 and core 802 via internal connections 820A, 820B, 820C, and 820D. The internal connections may be implemented as a bus, multiplexers, crossbar switches, and / or any other suitable connection technology.

[0206] Input and output to the processor 800 may be provided via a bus 816, which may include one or more conductors. The bus 816 may be communicatively coupled to one or more components of the processor 800, such as the controller 814, cache 810, and / or registers 812. The bus 816 may be coupled to one or more components of the system.

[0207] The bus 816 may be coupled to one or more external memories. The external memory may include a read-only memory (ROM) 832. The ROM 832 may be a masked ROM, an electronically programmable read-only memory (EPROM), or any other suitable technology. The external memory may include a random access memory (RAM) 833. The RAM 833 may be a static RAM, a backup static RAM, a dynamic RAM (DRAM), or any other suitable technology. The external memory may include an electrically erasable programmable read-only memory (EEPROM) 835. The external memory may include a flash memory 834. The external memory may include a magnetic storage device, such as a disk 836. In some embodiments, the external memory may be included in the system.

[0208] The present invention can be realized in any suitable form, including hardware, software, firmware or any combination of these. The present invention can be realized optionally at least in part as computer software running on one or more data processors and / or digital signal processors. The elements and components of embodiments of the present invention can be realized physically, functionally and logically in any suitable manner. In fact, the function can be realized in a single unit, in multiple units or as a part of other functional units. Like this, the present invention can be realized in a single unit, or can be physically and functionally distributed between different units, circuits and processors.

[0209] Although the present invention has been described in conjunction with some embodiments, it is not intended to be limited to the specific forms set forth herein. On the contrary, the scope of the present invention is limited only by the appended claims. In addition, although features may appear to be described in conjunction with specific embodiments, those skilled in the art will recognize that the various features of the described embodiments may be combined according to the present invention. In the claims, the term includes does not exclude the presence of other elements or steps.

[0210] In addition, although listed separately, multiple units, elements, circuits or method steps may be implemented by, for example, a single circuit, unit or processor. In addition, although individual features may be included in different claims, these features may be advantageously combined, and being included in different claims does not imply that the combination of features is not feasible and / or advantageous. In addition, including a feature in a claim category does not imply a restriction on the category, but rather indicates that the feature is equally applicable to other claim categories. In addition, the order of the features in the claims does not imply any specific order in which the features must work, and in particular the order of the individual steps in the method claims does not imply that the steps must be performed in this order. On the contrary, the steps may be performed in any suitable order. In addition, singular references do not exclude multiples. Therefore, references to "a", "an", "first", "second", etc. do not exclude multiples. The figure marks in the claims are provided only as clarification examples and should not be interpreted as limiting the scope of the claims in any way.

Claims

1. A device for generating a multi-channel audio signal, the device comprising: A receiver (101) is used to receive a data signal, wherein the data signal comprises: a downmixed audio signal for the multi-channel signal; upmix parameterized data for upmixing the downmix audio signal; a set of control data values ​​for adapting processing to generate an auxiliary signal for upmixing from the downmix audio signal; an artificial neural network (107) having an input node for receiving samples of the downmix audio signal and an output node arranged to provide samples of the auxiliary audio signal, the artificial neural network (107) further comprising a node arranged to include a contribution from the set of control data values; and A generator (105) is arranged to generate the multi-channel audio signal from the downmix signal and the auxiliary signal according to the upmix parameterization data.

2. The apparatus according to claim 1 , further comprising a filter bank (401) arranged to generate a frequency sub-band representation of the downmix audio signal; and wherein The samples of the downmix audio signal are sub-band samples of the frequency sub-band representation.

3. The device according to claim 2, wherein: The artificial neural network (107) is an artificial neural network of a first plurality of sub-band artificial neural networks, each sub-band artificial neural network of the first plurality of sub-band neural networks being arranged to generate sub-band samples for a sub-band subset of a frequency sub-band representation of the auxiliary audio signal.

4. The device according to claim 3, wherein: At least one control data value of the set of control data values ​​is processed by at least two artificial neural networks of the first plurality of sub-band artificial neural networks.

5. The device according to claim 3, wherein: At least one control data value of the set of control data values ​​is not processed by at least one artificial neural network of the first plurality of sub-band artificial neural networks.

6. The device according to claim 1, wherein: The artificial neural network (107) is trained by training data, the training data comprising at least one of the following: a training multi-channel audio signal, and a training downmix audio signal, and training upmix parameterized data generated from the training multi-channel audio signal; the training employs a cost function for comparing the training multi-channel audio signal with an upmix multi-channel signal, the upmix multi-channel signal being generated from the training downmix signal and from the generated auxiliary audio signal using the training upmix parameterized data.

7. An apparatus according to any preceding claim, wherein: The artificial neural network (107) is trained using a cost function and with training data having training input data, the training input data comprising a training downmix audio signal of a training multi-channel audio signal and a training set of control data values ​​for the training downmix audio signal, the cost function comprising a contribution indicative of a difference between a training auxiliary audio signal generated by the artificial neural network in response to the training data and a training residual signal for the training downmix audio signal.

8. The device according to claim 7, wherein: The cost function further comprises a contribution indicating a correlation between the auxiliary audio signal and the downmix of a multi-channel signal.

9. An apparatus according to any preceding claim, further comprising an upsampler for temporally upsampling the set of control data values, and wherein, The artificial neural network (107) is arranged to include contributions from upsampled control data values.

10. An apparatus for generating a data signal representing a multi-channel audio signal, the apparatus comprising: a downmixer arranged to generate a downmix audio signal by downmixing the multi-channel audio signal, the downmixer being further arranged to generate upmix parameterization data for upmixing the downmix audio; an artificial neural network arranged to generate a set of control data values ​​for an adaptation process to generate an auxiliary audio signal for upmixing from the downmix audio signal, the artificial neural network having an output node for providing the set of control data values ​​and an input node for receiving samples of at least one of the downmix audio signal and the multi-channel audio signal; A generator arranged to generate a data signal to comprise said downmix audio signal, said upmix parameterization data and said set of control data values.

11. The device according to claim 10, wherein: The set of control data values ​​provides a potential representation of at least one of the downmix audio signal and the multi-channel audio signal.

12. The device according to claim 10 or claim 11, wherein: The artificial neural network is trained by training data having training input data as samples of a training audio signal, the training audio signal being at least one of a multi-channel audio signal and a downmix of a multi-channel audio signal, and using a cost function, the cost function comprising a component for indicating a difference between the training audio signal and an output audio signal, the output audio signal being generated in accordance with a set of control data values ​​generated by the artificial neural network for the training audio signal.

13. The device according to claim 12, wherein: The output audio signal is the output of a synthetic artificial neural network having an input node for receiving the set of control data values ​​generated by the artificial neural network for the training audio signal, and the artificial neural network and the synthetic artificial neural network are jointly trained.

14. The device according to claim 12 or 13, wherein: The training audio signal and the output audio signal are downmixed audio channels.

15. A method for generating a multi-channel audio signal, the method comprising: receiving a data signal, wherein the data signal comprises: a downmixed audio signal for the multi-channel signal; upmix parameterized data for upmixing the downmix audio signal; a set of control data values ​​for adapting processing to generate an auxiliary signal for upmixing from the downmix audio signal; an artificial neural network (107) having an input node for receiving samples of the downmix audio signal and an output node for providing samples of the auxiliary audio signal, the artificial neural network (107) further comprising a node arranged to include a contribution from the set of control data values; and The multi-channel audio signal is generated from the downmix signal and the auxiliary signal according to the upmix parameterization data.

16. A method of generating a data signal representing a multi-channel audio signal, the method comprising: generating a downmix audio signal by downmixing the multi-channel audio signal, the downmixer being further arranged to generate upmix parameterization data for upmixing the downmix audio; an artificial neural network generating a set of control data values ​​for an adaptation process to generate an auxiliary audio signal for upmixing from the downmix audio signal, the artificial neural network having an output node for providing the set of control data values ​​and an input node for receiving samples of at least one of the downmix audio signal and the multi-channel audio signal; The data signal is generated to comprise the downmix audio signal, the upmix parameterization data and the set of control data values.

17. A computer program product comprising computer program code means adapted to perform all the steps of claim 15 or 16 when said program is run on a computer.

Images