Generation of multi-channel audio signal

Abstract

An audio apparatus comprises a receiver (101) arranged to receive a data signal comprising a downmix audio signal for a multi-channel audio signal, upmix parameterized data for upmixing the downmix audio signal, and upmix parameterized data. A subband generator (103) generates a frequency subband signal of a downmix audio signal, and a parameter generator (105) generates a set of upmix parameter values. The neural network arrangement (107, 401) includes a plurality of subband artificial neural networks (107, 401) that receive upmix parameter values and samples of at least one frequency subband signal. A subband artificial neural network (107, 401) generates subband samples for subbands of a frequency subband representation of a multi-channel audio signal.

Description

Technical Field

[0001] The present invention relates to the generation of multi-channel audio signals and in particular, but not exclusively, to the decoding of a stereo signal from a downmixed mono signal. Background Art

[0002] Spatial audio applications have become numerous and widespread and increasingly form at least a part of many audiovisual experiences. Indeed, new and improved spatial experiences and applications are constantly being developed, which leads to increased demands on audio processing and rendering.

[0003] For example, in recent years, virtual reality (VR) and augmented reality (AR) have received increasing interest and several implementations and applications are reaching the consumer market. In fact, devices are being developed for rendering experiences and for capturing or recording suitable data for such applications. For example, relatively low-cost devices are being developed to allow game consoles to provide a complete VR experience. It is expected that this trend will continue, and in fact will increase as the market for VR and AR reaches a considerable size in a short time frame. In the audio domain, a prominent area explores the reproduction and synthesis of realistic and natural spatial audio. The ideal goal is to produce a natural audio source so that the user cannot recognize the difference between the synthesized or original audio source.

[0004] A great deal of research and development effort has been focused on providing efficient and high quality audio encoding and audio decoding for spatial audio. Frequently used spatial audio representations are multi-channel audio representations, including stereo representations, and efficient encoding of such multi-channel audio has been developed based on downmixing the multi-channel audio signal to a downmix channel with fewer channels. One of the major advances in low bitrate audio coding is the use of parametric multi-channel coding, where the downmix signal is generated together with parametric data that can be used to upmix the downmix signal to recreate the multi-channel audio signal.

[0005] In particular, instead of conventional mid-side or intensity decoding, in parametric multi-channel audio decoding, the multi-channel input signal is downmixed to a lower number of channels (e.g., two to one), and the multi-channel image (stereo) parameters are extracted. The downmix signal is then encoded using a more conventional audio decoder (e.g., a mono audio encoder). The downmixed bitstream is multiplexed with an encoded multi-channel image parameter bitstream. The bitstream is then sent to a decoder, where the process is reversed. First, the downmix audio signal is decoded, after which the multi-channel audio signal is reconstructed guided by the encoded multi-channel image / upmix parameters.

[0006] An example of stereo decoding is described in E. Schuijers, W. Oomen, B. den Brinker, J. Breebaart, "Advances in Parametric Coding for High-Quality Audio", 114th AES Convention, Amsterdam, The Netherlands, 2003, Preprint 5852. In the described method, the downmixed mono signal is parameterized by exploiting the natural separation of the signal into three components (objects): transients, sinusoids and noise. More details are provided in "Low Complexity Parametric Stereo Coding", 116th AES, Berlin, Germany, 2004, Preprint 6073, which describes how parametric stereo can be implemented with low (decoder) complexity when combining parametric stereo with spectral band replication (SBR).

[0007] In the described method, decoding is based on the use of a so-called decorrelation process. The decorrelation process generates a decorrelation helper signal from the mono signal. In the stereo reconstruction process, both the mono signal and the decorrelation helper signal are used to generate an upmixed stereo signal based on the upmix parameters. Specifically, the two signals can be multiplied by a 2x2 matrix related to time and frequency with coefficients determined from the upmix parameters to provide an output stereo signal.

[0008] However, although parametric stereo (PS) and similar downmix encoding / decoding methods are a jump from traditional stereo and multi-channel decoding, the method is not optimal in all scenarios. In particular, known encoding and decoding methods tend to introduce some distortions, changes, artifacts, etc., which may introduce differences between the (original) multi-channel audio signal input to the encoder and the multi-channel audio signal recreated at the decoder. Typically, the audio quality may degrade and imperfect reconstruction of the multi-channel occurs. In addition, the data rate may still be higher than expected, and / or the complexity of the processing / resource usage may be higher than preferred.

[0009] A further problem is the high complexity and computational load on the decoder side, which it would be desirable to reduce, especially for a given audio quality.

[0010] Therefore, improved methods would be advantageous. In particular, methods allowing increased flexibility, improved adaptability, improved performance, increased audio quality, improved audio quality vs. data rate tradeoff, reduced complexity and / or resource usage, reduced computational load, facilitated implementation, and / or improved audio experience would be advantageous. Summary of the invention

[0011] Accordingly, the Invention seeks to preferably mitigate, alleviate or eliminate one or more of the above mentioned disadvantages singly or in any combination.

[0012] According to an aspect of the present invention, an audio device for generating a multi-channel audio signal is provided, the device comprising: a receiver for receiving an audio data signal, the audio data signal comprising: a downmix audio signal for the multi-channel signal; upmix parameterization data for upmixing the downmix audio signal; a subband generator for generating a set of frequency subband signals for a subband of the downmix audio signal; an artificial neural network arrangement comprising a plurality of subband artificial neural networks, each of the plurality of subband artificial neural networks being arranged to generate a subband for a subband of a frequency subband representation of the multi-channel audio signal a parameter generator arranged to generate, from the upmix parameterization data, a set of upmix parameter values ​​for a subband represented by the frequency subband of the multichannel audio signal; a generator for generating the multichannel audio signal from the subband samples of the subband of the multichannel audio signal; and wherein each subband artificial neural network comprises a set of nodes arranged to receive a set of upmix parameter values ​​and samples of at least one frequency subband signal from the set of frequency subband signals, the at least one frequency subband signal being a subband for which the subband artificial neural network generates subband samples of the multichannel audio signal.

[0013] In many embodiments, the method may provide an improved audio experience.For many signals and scenarios, the method may provide an improved generation / reconstruction of a multi-channel audio signal with improved perceived audio quality.

[0014] The method may provide a particularly advantageous arrangement which in many embodiments and scenarios may allow the possibility of facilitating and / or improving the use of artificial neural networks in audio processing (typically including audio encoding and / or decoding). The method may allow the advantageous use of artificial neural networks when generating a multi-channel audio signal from a downmix audio signal.

[0015] The method may provide an efficient implementation and may allow reduced complexity and / or resource usage in many embodiments.The method may in many cases allow reducing the data rate of data representing a multi-channel audio signal for use with a downmix signal.

[0016] Subband samples may span a specific time and frequency range.

[0017] The upmix parameterized data may include parameters (values) that relate the properties of the downmix signal to the properties of the multi-channel audio signal. The upmix parameterized data may include data indicating relative properties between channels of the multi-channel audio signal. The upmix parameterized data may include data indicating property differences between channels of the multi-channel audio signal. The upmix parameterized data may include data that is perceptually related to the synthesis of the multi-channel audio signal. The property may be, for example, a difference in phase and / or intensity and / or timing and / or correlation. In some embodiments and scenarios, the upmix parameterized data may represent abstract properties that cannot be directly understood by humans / experts (but may generally facilitate better reconstruction / lower data rates, etc.). The upmix parameterized data may include data including at least one of inter-channel intensity differences, inter-channel timing differences, inter-channel correlations, and / or inter-channel phase differences for channels of the multi-channel audio signal.

[0018] An ANN is a trained artificial neural network.

[0019] The artificial neural network may be a trained artificial neural network trained by training data comprising a training multi-channel audio signal; the training employing a cost function comparing the training multi-channel audio signal with a multi-channel signal generated by the artificial neural network arrangement. The artificial neural network may be a trained artificial neural network trained by training data comprising training data representing a range of related audio sources, the range of related audio sources comprising recordings of music, video, film, telecommunications, etc.

[0020] The audio device may specifically be an audio decoder device.

[0021] The subband generator may generate a frequency subband signal for each subband of the downmix audio signal. Each frequency subband signal is represented by a (subband) sample. Each subband artificial neural network has an input node that receives the (subband) sample of the frequency subband signal generated by the subband generator (and there may be more than one subband of the downmix audio signal).

[0022] Each subband artificial neural network generates an output for a subband of a multi-channel audio signal. Each subband artificial neural network specifically generates subband samples representing a multi-channel audio signal in one subband. A subband signal for a subband signal of a subband can be generated by each subband artificial neural network. The subband signal generated for a subband includes subband samples generated for the subband by the subband artificial neural network of the subband.

[0023] The subbands of the downmix audio signal generated by the subband generator may be the same as the subbands of the multi-channel audio signal generated by the subband artificial neural network. However, in some embodiments, they may be different. For example, multiple subbands of the downmix audio signal may be fed to a single subband artificial neural network, which generates subband samples for a single subband of the multi-channel audio signal.

[0024] Each sub-band artificial neural network may comprise a set of nodes arranged to receive samples for a sub-band for which the sub-band artificial neural network generates sub-band samples for the multi-channel audio signal.

[0025] In many embodiments, the sub-bands may have equal bandwidth.

[0026] The number of hidden layers in a sub-band artificial neural network can typically be in the range of 2-100.

[0027] The number of input nodes in a sub-band artificial neural network can typically range from 16 to 1024.

[0028] The number of output nodes in a sub-band artificial neural network can typically range from 1 to 1024.

[0029] The number of nodes in the hidden layer of a sub-band artificial neural network may typically range from 64 to 20482.

[0030] The number of values ​​in the upmix parameter set for each subband may typically be in the range of 1 to 6.

[0031] According to an optional feature of the invention, at least a first subband artificial neural network of the plurality of subband artificial neural networks comprises a node for receiving parameter values ​​of the upmix parameter set for other subbands than the subband of the subband artificial neural network.

[0032] This may provide a particularly efficient implementation and / or improved performance.

[0033] According to an optional feature of the invention, at least some parameters of the sets of upmix parameter values ​​for different sub-band artificial neural networks are identical.

[0034] This may provide a particularly efficient implementation and / or improved performance.

[0035] According to an optional feature of the invention, at least some parameters of the sets of upmix parameter values ​​for different sub-band artificial neural networks are different.

[0036] This may provide a particularly efficient implementation and / or improved performance.

[0037] According to an optional feature of the invention, the plurality of sub-band artificial neural networks for at least one sub-band comprises separate artificial neural networks for different channels of the multi-channel audio signal.

[0038] This may provide a particularly efficient implementation and / or improved performance.In particular, separate sub-band artificial neural networks may be provided for a left channel signal and a right channel signal of a stereo multi-channel audio signal.

[0039] According to an optional feature of the invention, the parameter generator is arranged to change the resolution of the upmix parameter set relative to the resolution of the upmix parameterization data to match the resolution of the processing of the multiple subband artificial neural network; the resolution of the processing of the multiple subband artificial neural network is one of a frequency resolution for the subbands of the multiple subband networks and a time resolution for the processing time interval.

[0040] In some embodiments, the upmix parameterization data may have a temporal resolution different from a processing time interval for at least one subband artificial neural network of the plurality of subband networks; and the parameter generator is arranged to modify the temporal resolution of the set of upmix parameter values ​​to match the processing time interval for the at least one subband artificial neural network.

[0041] In some embodiments the upmix parameterization data may have a frequency resolution different from a frequency resolution of a subband of the downmix audio signal; and the parameter generator is arranged to modify the frequency resolution of the set of upmix parameter values ​​to match the frequency resolution of the subband of the downmix audio signal.

[0042] According to an optional feature of the invention, the parameter generator comprises at least one artificial neural network having nodes receiving parameter values ​​for the upmix parameterization data and an output node providing the set of upmix parameter values ​​to a first subband artificial neural network of the plurality of subband artificial neural networks.

[0043] This may provide a particularly efficient implementation and / or improved performance.

[0044] The at least one artificial neural network may include a plurality of sub-band artificial neural networks. In some embodiments, one or more of the at least one artificial neural network may be common to a plurality of sub-band artificial neural networks generating samples of the multi-channel audio signal.

[0045] According to an optional feature of the invention, the plurality of subband artificial neural networks may comprise, at least for a first subband, at least two subband artificial neural networks, the at least two subband artificial neural networks generating samples of different components of the subband signal for the first subband.

[0046] This may provide a particularly efficient implementation and / or improved performance.

[0047] According to an optional feature of the invention, the plurality of sub-band artificial neural networks are trained by having training data of a training input audio signal comprising samples of an input multi-channel audio signal and using a cost function, the cost function comprising a component indicative of a difference between the training input audio signal and a multi-channel audio signal generated by the sub-band artificial neural network.

[0048] This may provide a particularly efficient implementation and / or improved performance. In many embodiments, this may provide a particularly efficient and high performance training.

[0049] According to an optional feature of the invention, the plurality of sub-band artificial neural networks are trained by having training data of a training input audio signal comprising samples of an input multi-channel audio signal and using a cost function, the cost function comprising a component indicative of a difference between upmix parameters for the input audio signal and upmix parameters for the multi-channel audio signal generated by the sub-band artificial neural network.

[0050] This may provide a particularly efficient implementation and / or improved performance. In many embodiments, this may provide a particularly efficient and high performance training.

[0051] According to an optional feature of the invention, at least one of the multiple subband artificial neural networks comprises: a first sub-artificial neural network having a node for receiving samples of frequency subband signals for a subband of the subband artificial neural network, and an output node for providing samples of a modified downmix audio signal; a second sub-artificial neural network having a node for receiving samples of frequency subband signals for a subband of the subband artificial neural network, and an output node for providing samples of an auxiliary audio signal; a third sub-artificial neural network having a node for receiving samples of the modified downmix audio signal, a node for receiving samples of the auxiliary audio signal, and a node for receiving a set of upmix parameter values ​​for a subband of the subband artificial neural network, the third sub-artificial neural network being further arranged to generate subband samples of subbands for a frequency subband representation of the multi-channel audio signal.

[0052] This may provide a particularly efficient implementation and / or improved performance.

[0053] According to an optional feature of the invention, the set of upmix parameters has a different number of parameters for at least two subbands.

[0054] This may provide a particularly efficient implementation and / or improved performance.

[0055] According to an optional feature of the invention, the upmix parameterization data provides parameterization data for sequential time intervals, and wherein at least a first subband artificial neural network of the plurality of subband artificial neural networks comprises a node for receiving parameter values ​​from the set of upmix parameter values ​​for another time interval in the sequential time intervals different from the time interval for which subband samples of the multi-channel audio signal are generated.

[0056] This may provide a particularly efficient implementation and / or improved performance.

[0057] According to an optional feature of the invention, the plurality of sub-band artificial neural networks are arranged to receive no further input data than: sub-band samples of the downmix audio signal and parameter values ​​generated from the upmix parameterisation data.

[0058] This may provide a particularly efficient implementation and / or improved performance.

[0059] According to an optional feature of the invention, there is provided a method for generating a multi-channel audio signal, the method comprising: receiving an audio data signal, the audio data signal comprising: a downmix audio signal for the multi-channel signal; upmix parameterization data for upmixing the downmix audio signal; generating a set of frequency subband signals for a subband of the downmix audio signal; each subband artificial neural network of a plurality of subband artificial neural networks generating subband samples for a subband of a frequency subband representation of the multi-channel audio signal; generating a set of upmix parameter values ​​for a subband of the frequency subband representation of the multi-channel audio signal from the upmix parameterization data; generating the multi-channel audio signal from the subband samples of the subband of the multi-channel audio signal; and wherein each subband artificial neural network comprises a set of nodes, the set of nodes being arranged to receive a set of upmix parameter values ​​and samples of at least one frequency subband signal of the set of frequency subband signals, the at least one frequency subband signal being a subband for which the subband artificial neural network generates subband samples of the multi-channel audio signal.

[0060] These and other aspects, features and advantages of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings, in which

[0062] Figure 1 shows some elements of an example of an audio device according to some embodiments of the present invention;

[0063] Figure 2 An example of the structure of an artificial neural network is shown;

[0064] Figure 3 An example of a node of an artificial neural network is shown;

[0065] Figure 4 shows some elements of an example of an audio device according to some embodiments of the present invention;

[0066] Figure 5 shows some elements of an example of an audio device according to some embodiments of the present invention;

[0067] Figure 6 shows some elements of an example of an apparatus for training an artificial neural network for an audio device according to some embodiments of the present invention; and

[0068] Figure 7 Some elements of a possible arrangement of a processor for implementing elements of an audio device according to some embodiments of the invention are shown. DETAILED DESCRIPTION

[0069] Figure 1 Some elements of an audio device according to some embodiments of the invention are shown.

[0070] The audio device comprises a receiver 101 arranged to receive a data signal / bitstream comprising a downmix audio signal which is a downmix of a multi-channel audio signal. The following description will focus on the case where the multi-channel audio signal is a stereo signal and the downmix signal is a mono signal, but it will be appreciated that the methods and principles described are equally applicable to multi-channel audio signals having more than two channels and to downmix signals having more than a single channel (although fewer channels than a multi-channel audio signal).

[0071] In addition, the received data signal includes upmix parameterized data for upmixing the downmix audio signal. The upmix parameterized data may specifically be a set of upmix parameters indicating the relationship between signals of different audio channels of a multi-channel audio signal (specifically, a stereo signal) and / or between the downmix signal and the audio channels of the multi-channel audio signal. Typically, the upmix parameters may indicate a time difference, a phase difference, a level / intensity difference and / or a similarity measure, such as correlation. Typically, upmix parameters are provided on a time-by-time and frequency-by-frequency basis (time-frequency slices). For example, new parameters may be provided periodically for a subband set. Parameters may specifically include inter-channel phase difference (IPD), overall phase expansion (OPD), inter-channel correlation (ICC), channel phase difference (CPD) parameters as known from parametric stereo coding (and from higher channel coding).

[0072] Typically, the downmix audio signal is encoded, and in a specific example, the receiver 101 includes a decoder that decodes the downmix audio signal (i.e., a mono signal). It should be understood that in the case where the received downmix audio signal is not encoded, a decoder may not be required and the decoder may be considered as an integral part of the receiver. Similarly, the receiver 101 may include functionality for extracting and decoding data representing the upmix parameters.

[0073] Conventionally, decoding of a signal such as a PS encoded stereo signal is based on generating a decorrelated signal from a downmix audio signal (particularly a mono signal), and then applying a (time and frequency dependent) 2x2 matrix multiplication to the samples of the downmix audio and the decorrelated signal, resulting in an output multichannel audio signal. The coefficients of the 2x2 matrix are determined from the upmix parameters of the upmix parameterized data. However, while this approach may be suitable for many applications, it is not ideal in all cases and tends to have suboptimal performance in some scenarios. Figure 1 The method uses a fundamentally different approach, which may provide improved performance and / or facilitate implementation in many embodiments and scenarios.

[0074] exist Figure 1 In the method, a receiver 101 is coupled to a subband generator 103, which is arranged to generate a plurality of frequency subband signals for a subband of a downmix audio signal. Thus, a subband representation of the downmix audio signal is generated using the subband generator 103 for generating subband samples for different frequency subbands, and thus, the subband generator 103 generates a plurality of subband samples for different subbands.

[0075] In particular, the subband generator 103 may include a filter bank arranged to generate a frequency subband representation of the downmixed audio signal. The filter bank may be a quadrature mirror filter (QMF) bank, or may be implemented, for example, by a fast Fourier transform (FFT), but it will be appreciated that many other filter banks and methods for dividing an audio signal into a plurality of subband signals are known and may be used. The filter bank may specifically be a complex-valued pseudo-QMF bank that generates, for example, 32 or 64 complex-valued subband signals.

[0076] In many embodiments, filter bank 501 is arranged to generate a subband signal set for subbands with equal bandwidth. In other embodiments, filter bank 401 may be arranged to generate a subband signal with subbands with different bandwidths. For example, a higher frequency subband may have a higher bandwidth than a lower frequency subband. In addition, subbands may be grouped together to form higher bandwidth subbands.

[0077] Typically, a sub-band may have a bandwidth in the range of 10 Hz to 10,000 Hz.

[0078] The audio device further comprises a parameter generator 105, which is arranged to generate a set of upmix parameters for a subband of the downmix audio signal from the received upmix parameterization data. In some embodiments, the parameter generator 105 may simply forward the received upmix parameters without modification, but may select appropriate upmix parameters and distribute them to other functional units. In other embodiments, the parameter generator 105 may process the received upmix parameter values ​​to generate new parameter values, e.g. by interpolation and / or upsampling.

[0079] The audio device further comprises an artificial neural network arrangement comprising a plurality of sub-band artificial neural networks 107, for the sake of clarity, Figure 1 Only one subband artificial neural network 107 of the plurality of subband artificial neural networks 107 is shown in FIG. In many embodiments, the artificial neural network arrangement may include one subband artificial neural network 107 for each subband generated by the subband generator 103. Each of the subband artificial neural networks 107 is arranged to generate subband samples for a subband of a frequency subband representation of the multi-channel audio signal. Each subband artificial neural network 107 comprises a set of output nodes, which generates samples for a subband of the multi-channel audio signal being reconstructed. The subband artificial neural network 107 has nodes arranged to receive subband samples of the downmix audio signal, and in particular, subband samples of one or more subbands corresponding to the frequencies of the subbands of the multi-channel audio signal are provided to the input nodes of the subband artificial neural network 107, and the subband artificial neural network 107 generates output samples for the one or more subbands. In addition, the subband artificial neural network 107 comprises a node that receives a set of upmix parameter values ​​for the subband from the parameter generator 105. In many embodiments, an individual subband artificial neural network 107 may receive a set of upmix parameters, which are upmix parameters received in upmix parameterization data and provided for a subband of the specific subband artificial neural network 107 .

[0080] In many embodiments, the subbands generated by the subband generator 103 and the subbands for which the subband artificial neural network 107 generates samples may be the same. There may be a direct correspondence between the subbands of the downmix audio signal and the subbands of the multi-channel audio signal. Specifically, there may be one subband artificial neural network 107 for each subband generated by the subband generator 103, and each of these subband artificial neural networks 107 may generate subband samples of the multi-channel audio signal for the same subband. In some embodiments, some subbands of the downmix audio signal may, for example, be combined to be processed by the same subband artificial neural network 107 (which may be considered equivalent to one subband having a combined bandwidth of the combined subbands). The subband artificial neural network 107 may in this case generate subband samples for the combined subband.

[0081] As will be described in more detail later, the subband artificial neural network 107 is trained to generate subband samples that reconstruct the multi-channel audio signal from the downmix audio signal and the upmix parameters. In the method, the downmix audio signal is divided into subbands accordingly, which are processed by the trained subband artificial neural network 107 that directly generates subband samples of the multi-channel audio signal.

[0082] The sub-band artificial neural network 107 is coupled to a signal generator 109 which generates a multi-channel audio signal from sub-band samples of sub-bands of the multi-channel audio signal.

[0083] The subband samples from the subband artificial neural network are fed to the signal generator 109, which proceeds to generate the reconstructed multi-channel audio signal. For example, in some embodiments where a subband representation of the multi-channel audio signal is desired (e.g., because subsequent processing is also subband-based), the subband samples can simply be output from the subband artificial neural network according to a specific structure or format. In many embodiments, the signal generator 109 includes functionality for converting the subband representation of the reconstructed multi-channel audio signal into a time domain representation. The signal generator 109 may specifically include a synthesis filter bank that performs the inverse operation of the subband generator 103 (and specifically, the filter bank of the subband generator 103), thereby converting the subband representation into a time domain representation of the multi-channel audio signal.

[0084] The generator may be specifically arranged to generate a frequency / subband domain representation of the multi-channel audio signal by processing a frequency or subband domain representation of the downmix audio signal and a frequency / subband domain representation of the auxiliary audio signal. Thus, the processing of the generator 105 may be a subband processing, such as, for example, a matrix multiplication performed on each subband on the subband samples of the downmix audio signal and the auxiliary audio signal generated by the corresponding subband artificial neural network.

[0085] The resulting subband / frequency domain representation may then be used directly, or may be converted to a time domain representation, for example using a suitable synthesis filter bank, which may in particular be applied by a separate synthesis filter for each channel.

[0086] An artificial neural network as used in the described functionality may be a network of nodes arranged in layers and each node holding a node value. Figure 2 An example of a portion of an artificial neural network is shown.

[0087] The node value for a given node can be calculated as a contribution including some or often all nodes from the previous layer of the artificial neural network. Specifically, the node value for a node can be calculated as a weighted sum of the node values ​​output by all nodes of the previous layer. Typically, a bias can be added, and the result can be affected by an activation function. The activation function provides the necessary parts of each neuron by typically providing nonlinearity. This nonlinearity and activation function provide a significant impact in the learning and adaptation process of the neural network. Therefore, the node value is generated based on the node value of the previous layer.

[0088] The artificial neural network may specifically include an input layer 201, which includes a plurality of nodes receiving input data values ​​for the artificial neural network. Therefore, the node values ​​of the nodes of the input layer may generally be directly the input data values ​​for the artificial neural network, and therefore may not be calculated based on other node values.

[0089] The artificial neural network may further include none, one or more hidden layers 203 or processing layers. For each of such layers, the node values ​​are typically generated based on the node values ​​of the nodes of the previous layer, and specifically, the weighted combination and added bias are followed by an activation function.

[0090] Specifically, Figure 3 As shown, each node (which may also be called a neuron) can receive input values ​​(from nodes in the previous layer) and thereby compute a node value as a function of these values. Typically, this involves first generating a value that is a linear combination of the input values, each of which is weighted by a weight:

[0091]

[0092] Where w refers to the weight, x refers to the node of the previous layer, and n refers to the index of the different nodes of the previous layer.

[0093] An activation function can then be applied to the resulting combination. For example, the node value l can be determined as:

[0094] l=f(k)

[0095] The function may be, for example, a sigmoid, a Tanh or a rectified linear unit (ReLU) function (as described in Xavier Glorot, Antoine Bordes, Yoshua Bengio Proceedings of the Fourtec International Conference on Artificial Intelligence and Statistics PML 15: 315-323, 2011):

[0096] f(k)=ReLU(f)=max(0,k)

[0097] Other commonly used functions include sigmoid function or tanh function. In many embodiments, multiple functions can be used to calculate node output or value. For example, both ReLU and Sigmoid functions can be combined using activation functions such as:

[0098] f(k)=ReLU(k)+σ(k)

[0099] Such operations can be performed by every node of the artificial neural network (usually except the input nodes).

[0100] The artificial neural network also includes an output layer 205, which provides the output from the artificial neural network, i.e. the output data of the artificial neural network are the node values ​​of the output layer. For the hidden / processing layers, the output node values ​​are generated by a function of the node values ​​of the previous layers. However, in contrast to the hidden / processing layers, where the node values ​​are generally not accessible or further used, the node values ​​of the output layer are accessible and provide the result of the operation of the artificial neural network.

[0101] Several different network structures and toolboxes have been developed for artificial neural networks, and in many embodiments, artificial neural networks can be based on adapting and customizing such networks. An example of a network architecture that may be suitable for the above applications is the WaveNet of van den Oord et al., which is described in Oord, Aaron van den, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. "Wavenet: A generative model for raw audio." arXiv preprint arXiv:1609.03499 (2016).

[0102] WaveNet is an architecture for synthesizing time-domain signals using dilated causal convolutions and has been successfully applied to audio signals. For WaveNet, the following activation functions are usually used:

[0103] z=tanh(W f，k *x)⊙σ(W g，k *x), where * represents the convolution operator, ⊙ represents the element-wise multiplication operator, σ(·) is the sigmoid function, k is the layer index, f and g represent the filter and gate, respectively, and W represents the weights of the learned artificial neural network. The filter product of the equation can generally provide a filtering effect, where the gated product provides a weighting of the result, which in many cases can effectively allow the contribution of a node to be reduced to essentially zero (i.e., it can allow or "cut off" a node that provides contributions to other nodes, thereby providing a "gate" function). In different cases, the gate function can cause the output of the node to be negligible, while in other cases it will substantially contribute to the output. Such a function can substantially help allow the neural network to learn and train efficiently.

[0104] In some cases, the artificial neural network can be further arranged to include additional contributions that allow the artificial neural network to be dynamically adapted or customized for specific desired properties or characteristics of the generated output. For example, a set of values ​​can be provided to adapt the artificial neural network. For example, a set of values ​​can be provided to adapt the artificial neural network. These values ​​can be included by providing contributions to some nodes of the artificial neural network. These nodes can be specific input nodes, but can generally be nodes of hidden layers or processing layers. Such adaptation values ​​can, for example, be weighted and added as a contribution to the weighted sum / correlation value for a given node. For example, for WaveNet, such adaptation values ​​can be included in the activation function. For example, the output of the activation function can be given as:

[0105] z=tanh(W f，k *x)⊙σ(W g，k *x+V g，k *y)

[0106] where y is a vector representing the adaptation values, and V represents the appropriate weights for these values.

[0107] The above description relates to a neural network approach that may be applicable to many embodiments and implementations. However, it should be understood that many other types and structures of neural networks may be used. Indeed, many different methods for generating neural networks have been and are being developed, including neural networks that use complex structures and processes different from those described above. The method is not limited to any specific neural network method, and any suitable method may be used without detracting from the invention.

[0108] exist Figure 1 In the audio device of the present invention, a single sub-band artificial neural network 107 is used to generate sub-band samples of multiple channels of a multi-channel audio signal in a given sub-band, and specifically, a single sub-band artificial neural network 107 is used to generate sub-band samples for two channels of a stereo signal. Therefore, the sub-band artificial neural network 107 can generate output samples for both the left channel and the right channel.

[0109] However, in many embodiments, a separate sub-band artificial neural network 107 may be provided for each channel of the multi-channel audio signal, and thus, a parallel structure and arrangement of the sub-band artificial neural networks 107 may be provided for each channel.

[0110] Figure 4 Shown with Figure 1 An example of an audio device corresponding to an audio device of , but having separate sub-band artificial neural networks 107, 401 for the left channel and the right channel, respectively. In this example, the sub-band artificial neural network 107 is arranged and trained to generate samples of the left signal of the stereo signal. In addition, the second sub-band artificial neural network 401 is arranged and trained to generate samples of the right signal of the stereo signal. Both the first sub-band artificial neural network 107 and the second sub-band artificial neural network 401 have input nodes that receive sub-band samples from the sub-band generator 103. Similarly, both the first sub-band artificial neural network 107 and the second sub-band artificial neural network 401 receive upmix parameters from the parameter generator 105, but in some embodiments, they may receive different upmix parameters. For example, one or more upmix parameters may specifically indicate the properties of one channel signal relative to the downmix audio signal, and such parameters may specifically be provided only to the sub-band artificial neural networks 107, 401 for the channels to which the upmix parameters are related.

[0111] In such embodiments, separate signal generators may be provided and applied to separate sub-band artificial neural networks 107, 401. Specifically, the signal generator 109 may receive output samples from the first sub-band artificial neural network 107, and the second and separate signal generator 403 may receive sub-band samples from the second sub-band artificial neural network 401.

[0112] In this method, the left channel and the right channel of the multi-channel audio signal are directly obtained from two pre-trained artificial neural networks that use the downmix signal and the decoded upmix parameters as input. The method is performed on a sub-band basis, so that separate artificial neural networks are provided for different sub-bands, and the multi-channel audio signal reconstruction and synthesis are achieved by a combination of multiple sub-band artificial neural networks, and the multiple sub-band artificial neural networks interact to provide direct generation of the multi-channel audio signal. The artificial neural network arrangement does not require or include any decorrelated signals or rotation / matrix multiplications to be performed (although in some cases, the artificial neural network may potentially lead to operations that may correspond to more extensive (and less obviously propagable) variants of such operations). It has been found that this method provides substantially improved reconstruction of multi-channel audio signals in many scenarios and embodiments. In addition, it can facilitate implementation and generally reduce the computational burden. For example, the use of a sub-band method can generally reduce complexity and resource usage, because a separate sub-band artificial neural network can generally be substantially smaller than, for example, a single artificial neural network required to generate samples of all frequencies for a multi-channel audio signal. Although more artificial neural networks may be used for sub-band processing, the size reduction that can be achieved is generally much greater than simple linear scaling, and therefore an overall complexity reduction requiring significantly fewer computations and operations can be achieved.

[0113] The subband artificial neural network 107 may specifically receive subband samples of the downmix audio signal and parameter values ​​determined from the subband generator 103 from the received upmix parameterized data, but generally no other input data may be provided. Thus, the method does not require any additional information, operations or data other than the information, operations or data of the encoded signal representing the multi-channel audio signal, and in particular does not require any data other than the subband samples and the upmix parameters.

[0114] In this arrangement each of the subband artificial neural networks receives subband samples for a subband of the subband artificial neural network and further all subband artificial neural networks are arranged to receive a set of upmix parameter values.

[0115] Each of the subband artificial neural networks generates subband samples for a subband subset of a frequency subband representation of the multi-channel audio signal, and typically generates subband samples (only) for the subband for which it receives input samples from the subband generator 103 .

[0116] In many embodiments, the apparatus comprises an artificial neural network for each subband of the frequency subband representation of the downmix audio signal generated by the subband generator 103. Thus, in many embodiments, the output samples for each subband of the subband generator 103 are fed to an input node of one subband artificial neural network, which then generates subband samples of the multi-channel audio signal for that subband. In many embodiments, the subband processing may thus be completely separated for each subband.

[0117] In this example, the generation of the multi-channel audio signal is therefore performed on a sub-band basis using separate and individual artificial neural networks in each sub-band. The individual artificial neural networks are trained to provide output samples for the sub-bands for which input sub-band samples are provided.

[0118] It has been found that this method provides a very advantageous generation of multi-channel audio signals, which allows extremely high quality reconstruction of the multi-channel audio signals. In addition, it can allow efficient operation with significantly reduced complexity and / or generally significantly reduced computational resource requirements. The sub-band artificial neural network tends to be substantially smaller than the single complete artificial neural network required for generating the entire signal. Typically, far fewer nodes, and possibly even fewer layers, are required for processing, resulting in a very large reduction in the number of operations and calculations required to implement the artificial neural network functions. Although more artificial neural networks are needed to cover all sub-bands, smaller artificial neural networks will generally result in a huge reduction in the total number of operations required, and therefore a huge reduction in overall computational resource requirements. In addition, in many scenarios, it can allow a more efficient learning process.

[0119] The subband arrangement may accordingly provide a computationally efficient method for allowing an artificial neural network to be implemented as an aid in the reconstruction of a multi-channel audio signal which has been encoded as a downmix audio signal with upmix parameterized data. The described system and method allow for the reconstruction of a high quality multi-channel audio signal and a generally significantly improved audio quality may be achieved compared to conventional methods. Furthermore, a computationally efficient decoding process may be achieved. The subband and artificial neural network based methods may also be compatible with other processes using subband processing.

[0120] In some embodiments, subband processing can be more flexible than strict subband by subband processing. For example, in some embodiments, each subband artificial neural network can receive subband samples not only from the subband itself, but also possibly for one or more other subbands. For example, in some embodiments, a subband artificial neural network for a subband can also receive samples of a downmixed audio signal from one or two neighboring / adjacent subbands. As another example, in some embodiments, one or more of the subband artificial neural networks can also receive input samples from one or more subbands including harmonics (or sub-harmonics) of the frequency for the subband. For example, a subband around a 500 Hz center frequency can also receive frequencies from a subband around a 1000 Hz center frequency. Such additional subbands having a specific relationship to the subbands of the subband artificial neural network can provide additional information, which can allow the generation of improved subband artificial neural networks for some audio signals.

[0121] In some embodiments, all sub-band artificial neural networks may have the same properties and dimensions. In particular, in many embodiments, all sub-band artificial neural networks may have the same number of input nodes and output nodes, and may have the same internal structure. This approach may be used, for example, in embodiments where all sub-bands have the same bandwidth.

[0122] However, in some embodiments, the sub-band artificial neural networks may comprise non-identical neural networks. In particular, in some embodiments, the number of input nodes for the sub-band artificial neural networks may be different for at least two of the artificial neural networks. Thus, in some embodiments, the number of input samples included in the determination of output samples may be different for different sub-bands and sub-band artificial neural networks.

[0123] In some embodiments, for some lower frequency sub-bands, the number of samples / input nodes may be higher than for some higher frequency bands. In fact, for increasing frequency, the number of samples / input nodes may decrease monotonically. The lower frequency sub-band artificial neural network may therefore be larger and consider more input samples than the higher frequency sub-band artificial neural network. This approach may, for example, be combined with sub-bands having different bandwidths, such as when the lower frequency sub-band may have a higher bandwidth than the higher frequency bandwidth.

[0124] Such an approach may provide an improved trade-off between achievable audio quality and computational complexity and resource usage in many scenarios. It may provide a tighter adaptation of the system to reflect the typical characteristics of audio, allowing more efficient processing.

[0125] In some embodiments, the subband artificial neural network may be used only for a subset of subbands of the downmix audio signal and / or the multi-channel audio signal. For other subbands, other methods may be applied, and in particular for one or more subbands, conventional methods of generating a decorrelated signal that is mixed with a mono downmix audio signal to generate a stereo signal may be applied. Thus, the described subband artificial neural network method may be applied only to some subbands.

[0126] In some embodiments, for some lower frequency bandwidths, the number of hidden layers can be higher than for some higher frequency bandwidths. The number of hidden layers can decrease monotonically for increasing frequencies. Such an approach can provide an improved tradeoff between achievable audio quality and computational complexity and resource usage in many scenarios.

[0127] In some embodiments, at least some of the upmix parameter values ​​of the set of upmix parameter values ​​are common to a plurality of subbands of the frequency subband representation of the downmix audio signal.

[0128] In some embodiments, the sub-band artificial neural networks are all provided with the same set of upmix parameter values. In some embodiments, the same set of upmix parameter values ​​may be provided to only some of the sub-band artificial neural networks. In particular, in some embodiments, at least one of the control data values ​​is processed by at least two synthesized sub-band artificial neural networks.

[0129] In many embodiments, using the same set of upmix parameter values ​​may provide improved efficiency and performance. It may generally reduce complexity and resource usage when generating the set of upmix parameter values. Furthermore, in many scenarios it may provide improved operation, and all available information provided by the parameter values ​​may be considered by each sub-band artificial neural network, and thus improved adaptation of the sub-band artificial neural networks may be achieved.

[0130] However, in some embodiments, different sets of upmix parameter values ​​may be provided to different sub-band artificial neural networks. Specifically, in some embodiments, at least one parameter value in the set of upmix parameter values ​​is not processed by at least one sub-band artificial neural network.

[0131] For example, the parameter generator 105 can generate a set of upmix parameter values, and different subsets of these parameter values ​​can be provided to different sub-band artificial neural networks. In other embodiments, some parameter sets can also be generated to include some parameter values, for example, provided manually or generated by analysis of the downmix audio signal. For example, harmonics or peaks can be detected in the downmix audio signal. Such data can, for example, be applied only to some of the synthetic sub-band artificial neural networks in the synthetic sub-band artificial neural network. For example, the detected peaks or harmonics can be indicated only to the synthetic sub-band artificial neural network in which they are detected. In some embodiments, such attributes and features can be generated on the encoder side and provided to the audio device as part of the data signal. Such received features can also be provided to the artificial neural network, that is, the sub-band artificial neural network can include an input node for receiving a value representing such an attribute.

[0132] In some embodiments, the encoder may alternatively or additionally generate upmix parameterization data in the form of data representing or describing properties of, for example, a downmix audio signal, for example, related to properties of one or more channels of a multi-channel audio signal. Indeed, in some embodiments, the encoder may include an artificial neural network that generates a set of parameter values ​​that provides a potential representation for the downmix audio signal, the potential representation being particularly suitable for upmixing the downmix audio signal to reconstruct the multi-channel audio signal, wherein the upmixing is performed by an artificial neural network arrangement as described herein. In this case, the encoder artificial neural network that generates the potential representation / parameter values ​​may be jointly trained with the subband artificial neural network 107.

[0133] Thus, in some embodiments, different sets of upmix parameter values ​​may be provided to different sub-band artificial neural networks. In many cases, this may include some parameter values ​​being the same and some parameter values ​​being different for different sub-band artificial neural networks.

[0134] As described above, in some embodiments the parameter generator 105 may generate sets of upmix parameters for different subband artificial neural networks 107 by selecting appropriate parameters from the received upmix parameterization data and providing these parameters directly to the appropriate subband artificial neural networks 107 .

[0135] For example, in many embodiments, the upmix parameterization data includes frequency and time related upmix parameters. For example, IPD, OPD, ICC, CPD parameters may be provided for different time-frequency tiles. In this case, for a given time interval, the upmix parameters provided in the upmix parameterization data for a frequency subband of one subband artificial neural network 107 may be compiled into an upmix parameter set, which is then fed to the subband artificial neural network 107 when processing the given time interval. Thus, when a given subband artificial neural network 107 generates subband samples for a given time interval, the parameter generator 105 may generate an upmix parameter set, the upmix parameter set comprising the upmix parameters provided for the time interval and for the subband. In addition, the subband artificial neural network 107 will receive subband samples of the downmix audio signal, thereby enabling it to process this input data to generate subband samples for the reconstructed multi-channel audio signal.

[0136] In some embodiments, each upmix parameter set may include parameter values ​​only for the subband for which the subband artificial neural network 107 generates samples, and thus only parameter values ​​only for the subband of the subband artificial neural network 107 are provided to the individual subband artificial neural network 107. However, in some embodiments, the upmix parameter set for one subband artificial neural network 107 may include parameter values ​​for other subbands than the subband for which the subband artificial neural network 107 generates samples. Therefore, in some embodiments, the subband artificial neural network 107 may include nodes for receiving parameter values ​​for other subbands than the subband of the subband artificial neural network.

[0137] In some embodiments, the parameter generator 105 may generate a set of upmix parameter values ​​for a given subband based on the received upmix parameterization data for the subband. However, one or more of the subband artificial neural networks may also include parameter values ​​for one or more other subbands in addition to the set generated for the subband of the subband artificial neural network, i.e., the input to the subband artificial neural network may have an input node that receives subband samples for a subband for which the subband artificial neural network does not generate any samples.

[0138] As a specific example, in many embodiments, each sub-band artificial neural network may receive as input parameter values ​​not only from the sub-band for which the sample set was generated, but also from neighboring sub-bands.

[0139] Such methods may generally allow the generation of an improved set of upmix parameter values, which may result in improved audio quality. In particular, it has been found that taking surrounding subbands into account may allow the set of upmix parameter values ​​to better reflect the temporal resolution of the downmix audio signal / multi-channel audio signal. It has been found that such methods may in particular allow a better representation of temporal peaks.

[0140] In some embodiments, one or more of the subband artificial neural networks may also be fed subband samples outside of the time interval for which the subband artificial neural network generated samples of the multi-channel audio signal. The subband artificial neural network may include an input node that receives subband samples from outside of the current time interval.

[0141] Specifically, the processing of the audio device can be performed on a frame-by-frame basis, wherein a time interval / frame of the received downmix audio signal is processed to generate output samples of the multi-channel audio signal for the time interval / frame. Therefore, for each frame, the sub-band generator 103 generates sub-band samples, the parameter generator 105 generates parameter values ​​for the sub-band for the frame, and the sub-band samples and parameter values ​​are fed to the sub-band artificial neural network, which generates sub-band samples of the multi-channel audio signal for the frame / time interval of the multi-channel audio signal.

[0142] Hence, in particular, each subband artificial neural network operates in blocks, wherein each operation of generating a set of output samples from a set of input samples corresponds to a time interval of the downmix audio signal / multi-channel audio signal for which an output sample of the multi-channel audio signal is generated.

[0143] In some embodiments, in addition to the parameter values ​​given for the subband, one or more of the subband artificial neural networks may also receive parameter values ​​for another time interval, such as typically from one or more adjacent time intervals. For example, in some embodiments, one or more of the subband artificial neural networks may also include parameter values ​​for the previous and next time intervals.

[0144] In such examples, the upmix parameterization data accordingly provides parameters for a plurality of different sequential time intervals. For a given time interval, the subband artificial neural network 107 is provided with parameter values ​​for the time interval of the multi-channel audio signal for which the subband samples are generated. However, in addition, in some embodiments, one or more of the subband artificial neural networks 107 further comprises nodes for receiving parameter values ​​for other time intervals of the sequential time intervals, the other time intervals being different from the time interval for which the subband samples of the multi-channel audio signal are (currently) generated.

[0145] In some embodiments, the subband samples provided to the subband artificial neural network 107 may be only for the subband for which the subband artificial neural network 107 generates samples, and thus the subband samples for the subbands of an individual subband artificial neural network 107 are only provided to the subband artificial neural network 107. However, in some embodiments, the subband samples for one subband artificial neural network 107 may include subband samples for other subbands than the subband for which the subband artificial neural network 107 generates samples. Therefore, in some embodiments, the subband artificial neural network 107 may include a node for receiving subband samples for other subbands than the subband of the subband artificial neural network.

[0146] As a specific example, in many embodiments, each subband artificial neural network may receive as input not only subband samples from the subband for which the set of upmix parameter values ​​is generated, but also subband samples from neighboring subbands.

[0147] Such an approach may generally result in improved audio quality. In particular, it has been found that taking surrounding subbands into account may allow the generated subband samples to better reflect the temporal resolution of the downmix audio signal / multi-channel audio signal. It has been found that such an approach may in particular allow a better representation of temporal peaks.

[0148] Each subband artificial neural network operates in processing time intervals, wherein each operation of generating a set of output samples from a set of input samples corresponds to a time interval of the downmix audio signal / multi-channel audio signal for which an output sample of the multi-channel audio signal is generated.

[0149] In some embodiments, in addition to the subband samples given for the time interval, one or more of the subband artificial neural networks may also receive subband samples for another time interval, for example, typically from one or more adjacent time intervals. For example, in some embodiments, one or more of the subband artificial neural networks may also include subband samples for the previous and next time intervals.

[0150] Typically, the upmix parameters received in the upmix parameterization data will have a different time and frequency resolution than the subband domain downmix audio signal and the subband and processing time interval of the subband artificial neural network 107. In many embodiments, the parameter generator 105 may be arranged to adapt the received upmix parameters to generate a set of upmix parameter values ​​that matches the time and frequency resolution of the processing of the subband artificial neural network 107.

[0151] In many embodiments, the parameter generator 105 is accordingly arranged to change the resolution of the set of upmix parameters relative to the resolution of the upmix parameterization data to match the resolution of the processing of the multiple subband artificial neural network 107. The change of resolution may be in the frequency domain and / or the time domain and may be performed to align the upmix parameterization data with the frequency resolution of the subbands and / or the time resolution of the processing time intervals for the multiple subband networks.

[0152] In some embodiments, the change in resolution may effectively be a resampling of the received parameter values ​​to match the time and frequency resolution of the subband processing. For example, in some embodiments, linear interpolation may be applied to each parameter to generate sample values ​​of the parameter for a time and frequency interval corresponding to the subband processing. For example, if the upmix parameterization data includes parameter values ​​for two frequencies corresponding to two adjacent subbands greater than the subband processing, the parameter values ​​for processing the subband may be found by simple interpolation between the parameter values ​​included in the upmix parameterization data. Similarly, if the parameter values ​​of the upmix parameterization data are for a time interval greater than the processing time interval, then interpolation may be applied to generate higher resolution parameter values ​​for the upmix parameter set. It will be appreciated that many different methods for resampling (both for increasing and decreasing resolution) are known to those skilled in the art, and any suitable method may be applied.

[0153] In some embodiments, the parameter generator 105 may advantageously include one or more artificial neural networks. In some embodiments, the parameter generator 105 may include a single artificial neural network that generates a set of upmix parameter values ​​for all subband artificial neural networks 107. However, in many embodiments, the parameter generator 105 may include artificial neural networks for multiple and typically all subband artificial neural networks 107. Thus, in some embodiments, the parameter generator 105 may include one artificial neural network for each subband artificial neural network 107.

[0154] Using a trained artificial neural network to generate a set of upmix parameters may provide improved operation and performance in many scenarios, and in particular using a sub-band artificial neural network to generate a set of upmix parameters may provide improved performance while keeping complexity and computational resource usage low.

[0155] In some embodiments, the upmix parameter set may include the same number of parameters for each subband artificial neural network 107, and each subband artificial neural network 107 may include the same number of nodes that receive contributions from the upmix parameters. For example, in some scenarios, each parameter set may include a set of complementary parameters for the subband of the subband artificial neural network 107 included in the upmix parameterization data. For example, an upmix parameter set including IID, IPD and ICC parameters may be provided for each subband and subband artificial neural network 107. Therefore, in such an embodiment, each subband artificial neural network 107 may receive one IID, one IPD and one ICC parameter, which reflects the upmix for that subband.

[0156] However, in other embodiments, the upmix parameter set has a different number of parameters for at least two subbands. For example, in some embodiments, for some subbands, only one or two of the IID, IPD and ICC parameters may be included, while for other subbands, all parameters may be provided. This may, for example, reflect that for some frequency ranges, some parameters are more relevant than other parameters.

[0157] As another example, in some embodiments, some subbands may have different sizes and the upmix parameterization data may include more upmix parameters for some subbands than for other subbands. The parameter generator 105 may, for example, generate more parameters for some subbands (e.g., for different frequency subranges within each subband) than for other subbands. The subband artificial neural network 107 that covers a larger bandwidth and for which more upmix parameters are generated may therefore be arranged to have more nodes that receive input from the upmix parameters than other subband artificial neural networks 107.

[0158] As another example, in many embodiments, the update rate for the upmix parameters may be different for different frequency ranges, and therefore, more parameter values ​​may exist (or be generated by the parameter generator 105) for some subbands than for other subbands. For example, for some embodiments, one set of upmix parameters may be provided for some subbands, while for other subbands, multiple sets may be provided for different times within a processing interval. The subband artificial neural networks 107 for these subbands may be arranged with different nodes for including such parameter values ​​in the determination of subband samples of the multi-channel audio signal.

[0159] Such methods may generally allow for improved reconstruction of multi-channel audio signals, wherein processing may be more accurately adapted to different properties for different frequency ranges.

[0160] In many embodiments, each subband may be processed by one subband artificial neural network 107 that generates samples for all subbands of the multi-channel audio signal. However, in some embodiments, a subband may include more than one subband artificial neural network 107 that generates subband samples for different parts of the subband multi-channel audio signal (since in practice one or more subbands may not include a subband artificial neural network).

[0161] For example, in some embodiments, the downmix audio signal subband samples and the set of upmix parameter values ​​for a given subband may be provided to two (or more) subband artificial neural networks 107, which generate subband samples for different parts of the subbands of the multi-channel audio signal.

[0162] The two subband artificial neural networks 107 for a given downmix audio signal subband may for example generate signals for different time intervals, e.g. one subband artificial neural network 107 may generate subband samples for the first half of the processing time interval and the other subband artificial neural network 107 may generate subband samples for the second half of the processing time interval.

[0163] In other embodiments, one subband artificial neural network 107 may, for example, generate subband samples for one sub-frequency range, and another subband artificial neural network 107 may generate samples for another sub-frequency range of the subband. For example, such methods may be particularly suitable for situations where the multi-channel audio signal may include, for example, a specific tonal component at a specific frequency. One of the subband artificial neural networks 107 may be trained to accurately reflect such a tonal component when such a tonal component is present, while the other subband artificial neural networks 107 do not need to be compromised by being trained to generate subband samples that do not need to reflect such a tonal component.

[0164] In such a scenario, the individual sub-band artificial neural networks may be trained to provide output samples specifically for the corresponding parts of the generated sub-band signals. For example, a sub-band artificial neural network arranged to generate sub-band samples for the first half of the processing time interval will be specifically trained based on a comparison of the generated samples with samples of the first half of the original multi-channel audio signal. Similarly, a sub-band artificial neural network 107 trained for a particular frequency interval of a sub-band will be trained based on the multi-channel audio signal within that frequency interval.

[0165] In many embodiments, the use of such multiple sub-band artificial neural networks 107 within each (downmix audio signal) sub-band may provide improved performance. It may generally allow the generation of an improved multi-channel audio signal that may more closely correspond to the original multi-channel audio signal. In particular, despite the use of more sub-band artificial neural networks, this may also allow reduced complexity / computational resources in many embodiments, as these may generally each have a much lower complexity and have fewer computational requirements.

[0166] In some embodiments, one or more sub-band artificial neural networks 107 in the sub-band artificial neural network 107 may be formed by including a plurality of sub-band artificial neural networks. Figure 5 As shown, the sub-band artificial neural network 107 can be formed by three sub-artificial neural networks. In this case, the sub-band artificial neural network 107 includes two sub-artificial neural networks 501, 503, both of which receive sub-band samples of the downmixed audio signal (corresponding to the sub-band artificial neural network 107 with two input nodes for each sub-band sample). The output nodes of these two sub-artificial neural networks are also input nodes of the third sub-artificial neural network 505, and the third sub-artificial neural network 505 also has an input node for receiving an upmix parameter set. The output node of the third sub-artificial neural network is the output node of the sub-band artificial neural network 107.

[0167] This arrangement has been found to be particularly efficient and provides high quality reconstruction of the multi-channel audio signal. The method may also allow sub-training, wherein in particular the first sub-artificial neural network 501 and the second sub-artificial neural network 503 may be individually and separately adapted to provide the desired results. This has been found to be particularly advantageous in many scenarios and for many signals.

[0168] Specifically, in some embodiments, the first sub-artificial neural network 501 can be trained to provide, for example, mono to mono processing, which provides a modified mono signal that is particularly suitable for upmixing. In addition, the second sub-artificial neural network 503 can be trained to provide a decorrelated or residual signal for a mono to mono downmix audio signal. The third sub-artificial neural network can be trained to provide a reconstructed multi-channel audio signal. The third sub-artificial neural network can be trained, for example, by end-to-end training, which is based on comparing the original multi-channel audio signal with the reconstructed multi-channel audio signal, and the reconstructed multi-channel audio signal is based on the first sub-artificial neural network 501 and the second sub-artificial neural network 503 having a configuration determined by a previous separate training.

[0169] In this approach, the first sub-artificial neural network may typically be relatively small, since the expected temporal distortion is typically small. The second sub-artificial neural network may be relatively large, especially at low frequencies, since such an artificial neural network can better ensure proper decorrelation. The third sub-artificial neural network will tend to be relatively small. Overall, reduced complexity and reduced computational resources can typically be achieved.

[0170] Artificial neural networks are adapted to specific purposes through a training process, which is used to adapt / adjust / modify the weights and other parameters (e.g., bias) of the artificial neural network. It should be understood that many different training processes and algorithms are known for training artificial neural networks. Typically, training is based on a large training set, in which a large number of examples of input data are provided to the network. In addition, the output of the artificial neural network is typically (directly or indirectly) compared with an expected or ideal result. A cost function can be generated to reflect the expected results of the training process. In a typical scenario known as supervised learning, the cost function typically represents the distance between the prediction and the ground truth for specific input data. Based on the cost function, the weights can be changed, and by re-iterating the process for modifying the weights, the artificial neural network can be adapted to a state where the cost function is minimized.

[0171] In more detail, during the training step, the neural network can have two different information flows from input to output (forward pass) and from output to input (backward pass). In the forward pass, as described above, the data is processed by the neural network, and in the backward pass, the weights are updated to minimize the cost function. Typically, this backward propagation follows the gradient direction of the cost function pattern. In other words, by comparing the predicted output with the ground truth for a batch of data input, the direction in which the cost function is minimized can be estimated and propagated backward by updating the weights accordingly. Other methods for training artificial neural networks include, for example, the Levenberg-Marquardt algorithm, the conjugate gradient method, and Newton's method.

[0172] In the present case, the training may specifically include a training set including a possibly large number of multi-channel audio signals. In some embodiments, the training data may be a multi-channel audio signal in a time segment corresponding to a processing time interval of the trained artificial neural network, for example, the number of samples in the training multi-channel audio signal may correspond to the number of samples corresponding to the input nodes of the trained artificial neural network. Each training example may therefore correspond to an operation of the trained artificial neural network. However, typically, a batch of training samples is considered for each step to speed up the training process. In addition, many upgrades to the gradient descent are also possible to speed up convergence or avoid local minima in the cost function situation.

[0173] For each training multi-channel audio signal, the training processor may perform a downmix operation to generate a downmix audio signal and corresponding upmix parameterized data. Thus, the encoding process applied to the multi-channel audio signal during normal operation may also be applied to the training multi-channel audio signal, thereby generating downmix and upmix parameterized data.

[0174] Additionally, in some embodiments, the training processor may generate a residual signal reflecting the difference between the downmix audio signal and the multi-channel audio signal, or more generally representing parts of the multi-channel audio signal that are not properly represented by the downmix audio signal. For example, in many embodiments, the training processor may generate a downmix signal and may additionally generate a residual signal that, when upmixed based on the upmix parameterization data, will generate a (more) accurate multi-channel audio signal to be reconstructed. Additionally, the training processor may generate upmix parameters.

[0175] Specifically, for stereo multi-channel audio signals, the training processor can use a parametric stereo scheme (e.g., according to a suitable standardization method). This encoding will apply a frequency- and time-related matrix operation, such as a rotation operation, to the input stereo signal to generate a downmix signal and a residual signal. For example, a 2x2 matrix multiplication / complex value multiplication is usually applied to the input stereo signal, such as to substantially align one of the rotated channel signals to have a maximum signal value. The channel can be used as a mono signal, and the rotation is usually performed on a frame basis. The rotation value can be stored as part of the upmix parameterized data (or the parameters allowing determination thereof can be included in the upmix parameterized data). Therefore, in a synthesis device, an opposite rotation can be performed to reconstruct a stereo signal. The rotation of the stereo signal causes another stereo signal, in which one of the channels is therefore aligned with the maximum intensity. Another channel is usually discarded in a parametric stereo encoder to reduce the data rate. In conventional parametric stereo decoding, a decorrelated signal is usually generated at a decoder and used for an upmix process. In the current training method, this second signal may be used as a residual signal for the downmix, since it may represent the information discarded in the encoder and therefore it represents the ideal signal to be reconstructed in the decoder as part of the upmix process.

[0176] Hence, in some embodiments, the training processor may generate a training downmix signal and / or a training residual signal and / or training upmix parameters from a training multi-channel audio signal.

[0177] Furthermore, the training processor may continue to generate subbands for the generated downmix signal (and potentially for the residual signal if these subbands are used). Similarly, for methods where the parameter generator 105 processing is not based on an artificial neural network but is a predetermined operation (e.g., selection or simple interpolation), the training processor may further continue to generate a set of upmix parameters.

[0178] Thus, the training processor may generate a training data set comprising a subband downmix audio signal and a subband set of upmix parameters, and in particular, it may comprise the same functionality as the encoder and decoder functionality (including the receiver 101, the subband generator 103 and, in some cases, the functionality of the parameter generator 105), which results in samples and a set of upmix parameters to be generated by the audio device and provided to the subband artificial neural network 107. Based on these input values, the subband artificial neural network 107 may then proceed to generate subband samples for the multi-channel audio signal. These may be converted to the time domain to result in a reconstructed multi-channel audio signal. This reconstructed multi-channel audio signal may then be compared with the original training multi-channel audio signal as part of a cost function, which may then be used to adapt the subband artificial neural network 107.

[0179] Accordingly, the training may be sub-band training, where sub-band data is generated and applied to separate sub-band artificial neural networks 107, and where the outputs of the sub-band artificial neural networks 107 are combined into a multi-channel audio signal which may be evaluated by a cost function.

[0180] In case the parameter generator 105 also comprises one or more artificial neural networks, the generated upmix parameterization data may also be applied to the parameter generator 105, wherein the generation is based on the currently configured upmix parameter set. In this case, the training, and in particular the updating, may include the artificial neural network of the parameter generator 105 as well as the sub-band artificial neural network.

[0181] Furthermore, in some embodiments, the cost function may include a contribution reflecting how close the upmix parameters generated by the artificial neural network of the parameter generator 105 are to the original training parameters generated by the training processor. In some embodiments, the parameter generator 105 may be trained separately from the sub-band artificial neural network 107 and the cost function may be used solely based on comparing the generated parameter values ​​with the input training upmix parameterization data.

[0182] However, in many embodiments, the artificial neural network of the parameter generator 105 may be jointly trained with the subband artificial neural network 107, and the cost function may in many cases include both: a contribution indicative of the difference between the input training multi-channel audio signal and the reconstructed multi-channel audio signal, and a contribution indicative of the difference between the upmix parameters for the input training multi-channel audio signal and the upmix parameters generated by the artificial neural network of the parameter generator 105.

[0183] In an embodiment where one or more of the sub-band artificial neural networks 107 are divided into sub-artificial neural networks, for example Figure 5In the example of , the training of the first sub-artificial neural network 501 and / or the second sub-artificial neural network 503 may be trained separately and before the training of the third sub-artificial neural network 505. For example, the first sub-artificial neural network 501 may be trained using the generated training downmix audio signal and comparing the obtained downmix audio signal therewith.

[0184] Similarly, the second sub-artificial neural network 503 may be trained based on being fed with sub-band samples of the training downmix audio signal and a cost function based on a comparison of the resulting output with the generated training residual signal.

[0185] A training method can be used in which outputs from a neural network operation are determined based on training signals, and a cost function is applied to determine a cost value for each training signal and / or for a combined set of signals (e.g., determining an average cost value for the training set). The cost function can include various components.

[0186] Typically, the cost function will include at least one component reflecting how close the generated signal is to a reference signal, the so-called reconstruction error. In some embodiments, the cost function will include at least one component reflecting how close the generated signal is to a reference signal from a perceptual viewpoint.

[0187] For example, in some embodiments, the multi-channel audio signal generated by the sub-band artificial neural network (and optionally, any artificial neural network of the parameter generator 105) for a given training multi-channel audio signal can be compared with the original training multi-channel audio signal. A cost function contribution reflecting the difference between these items can be generated. This process can be generated for all training sets to generate a total cost function. In addition, the method can be applied separately or jointly in each sub-band. In this case, the cost function can represent the difference between the sub-band of the reconstructed multi-channel audio signal and the sub-band of the original multi-channel audio signal. For example, using independent sub-band artificial neural networks, and assuming that perception is not considered, training of separate sub-band artificial neural networks (e.g., using a cost function of the RMSE type for each sub-band) is feasible.

[0188] exist Figure 6 This example is illustrated in for a stereo multi-channel audio signal. Figure 6, is an example of a training setup that specifically trains the left channel subband artificial neural network 107 and the right channel subband artificial neural network 401 (and in some cases, one or more artificial neural networks included in the parameter generator 105 may also be trained jointly with the subband artificial neural networks using such a training setup). It should be understood that in scenarios where other artificial neural networks may be present, for example, if other artificial neural networks are used in the encoder to generate upmix parameterization data (e.g., as a potential representation of the downmix audio signal), such artificial neural networks may be added to the training setup shown for joint training.

[0189] In this example, the training processor 601 may receive a training multi-channel audio signal, which in a specific example is a stereo signal. In an example, the multi-channel audio signal may be received as a sub-band signal, and the following description will focus on an implementation in a single sub-band. The same method may be reused for other sub-bands.

[0190] For a given training signal, the downmixer 603 performs a downmix operation to generate a training downmix signal, which in the specific example is a training mono audio signal. The downmix audio signal is input to the sub-band artificial neural network 107 .

[0191] In addition, the multi-channel audio signal is fed to a parameter estimator 605, which proceeds to generate upmix parameters, just as in the encoder. In this example, the upmix parameters are typically IID / IPD / ICC parameters generated, for example, by an analysis function applied to the input training multi-channel audio signal, specifically, these parameters are generated in the same way as they are in an encoder (e.g., a conventional encoder). These parameters are optionally quantized and encoded / decoded in an emulator 607 to generate upmix parameters, just as they are input to the parameter generator 105.

[0192] The parameter generator 105 and the sub-band artificial neural network 107, 401 may then proceed to reconstruct the stereo multi-channel audio signals l', r'.

[0193] The reconstructed signals l', r' are fed to a comparator 609 which proceeds to generate a cost value based on a cost function which may include a contribution indicating the difference between the original signal and the reconstructed signal. In many embodiments, the cost function may include a contribution reflecting how closely the upmix parameters generated by the parameter generator 105 match the upmix parameters generated by the parameter estimator.

[0194] It should be understood that many different methods can be used to determine the cost value reflecting the difference between the signals. For example, the correlation can be performed using a cost value having a monotonically decreasing value for the increased correlation value. As another example, the two signals can be subtracted from each other, and the power measurement for the difference signal can be used as the cost value. It should be understood that many other methods are available and can be used. As a specific example, a training process can be applied, which has the goal of minimizing the distance l-l' and r-r', and recovering the upmix parameters (e.g., IID / IPD / ICC) as close as possible. For a given frame (l, r), the (traditional) upmix parameters IID (level difference between left and right per band), ICC (correlation between left and right per band) and IPD (phase difference between left and right) can be determined, followed by energy-reserved downmix m. Then, the (optional) artificial neural network of the parameter generator 105 and the left and right subband artificial neural network 107 can be jointly trained to minimize the reconstruction errors l-l' and r-r' balanced with the recovery of the upmix parameters. This means that the loss function will be a combination of signal reconstruction error and PS parameter recovery. Specifically:

[0195] Loss = d reconstruction +α·d stereo

[0196] Where α is a parameter for adjusting the balance between signal reconstruction and stereo image reconstruction, and where:

[0197] d reconstruction =d(l,l′)+d(r,r′)

[0198] Or, usually:

[0199]

[0200] and:

[0201]

[0202] where iid(l,r) represents the intensity difference between left and right, possibly in the logarithmic domain for perceptual matching, and icc(l,r) represents the complex-valued (to include phase information) correlation between the left and right signals, possibly in the normalized domain for perceptual matching. Since the signal reconstruction error will already ensure that the IID is approximately restored, the first part of the stereo image loss can be discarded.

[0203] Thus, in this example, the cost function generates a cost value that reflects how closely the generated multi-channel audio signal matches the corresponding training multi-channel audio signal.

[0204] Based on the cost value, the training processor 601 can adapt the weights of the artificial neural network. For example, a back propagation method can be used. Specifically, the training processor 601 can adjust the weights of both the sub-band artificial neural network 107 and the artificial neural network of the parameter generator 105 based on the cost value. For example, given the derivative of the weight relative to the cost function (which represents the slope), the weight value is modified to move in the opposite direction of the slope. For simple / minimal cases, in the case of back propagation with a single data input, reference can be made to the training of a perceptron (a single neuron).

[0205] This process may be iterated until the artificial neural network is considered trained. For example, a predetermined number of iterations of training may be performed. As another example, training may continue until the weights change by less than a predetermined amount. It is also very common to implement a validation stop where the network is again tested for a validation metric and stops when the expected results are achieved.

[0206] The artificial neural network can (further) be trained using training data that does not directly represent the audio source / signal, but conveys relevant and similar meaningful information. A specific example is to include text-based training data. Training the artificial neural network based on text can allow the network to further improve its understanding of language, and therefore improve audio reconstruction. For example, by coupling text and audio, it is easier to predict word sequences than simply using one modality. The same applies to video and audio streams (e.g., in the example of lip sync or reading).

[0207] The audio device may be implemented in one or more appropriately programmed processors. In particular, the artificial neural network may be implemented in one or more such appropriately programmed processors. Different functional blocks, in particular the artificial neural network, may be implemented in separate processors and / or may be implemented, for example, in the same processor. Examples of suitable processors are provided below.

[0208] Figure 7 is a block diagram illustrating an example processor 700 according to an embodiment of the present disclosure. The processor 700 may be used to implement one or more processors that implement an apparatus or elements thereof as previously described (including, among other things, one or more artificial neural networks). The processor 700 may be any suitable processor type, including, but not limited to, a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), where the FPGA has been programmed to form a processor, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), where the ASIC has been designed to form a processor, or a combination thereof.

[0209] Processor 700 may include one or more cores 702. Core 702 may include one or more arithmetic logic units (ALUs) 704. In some embodiments, core 702 may include a floating point logic unit (FPLU) 706 and / or a digital signal processing unit (DSPU) 708 in addition to or in lieu of ALU 704.

[0210] The processor 700 may include one or more registers 712 communicatively coupled to the core 702. The registers 712 may be implemented using dedicated logic gates (e.g., flip-flops) and / or any memory technology. In some embodiments, the registers 712 may be implemented using static memory. The registers may provide data, instructions, and addresses to the core 702.

[0211] In some embodiments, the processor 700 may include one or more levels of cache memory 710 communicatively coupled to the core 702. The cache memory 710 may provide computer readable instructions to the core 702 for execution. The cache memory 710 may provide data for processing by the core 702. In some embodiments, the computer readable instructions may have been provided to the cache memory 710 by a local memory (e.g., a local memory attached to the external bus 716). The cache memory 710 may be implemented with any suitable cache memory type, for example, a metal oxide semiconductor (MOS) memory, such as a static random access memory (SRAM), a dynamic random access memory (DRAM), and / or any other suitable memory technology.

[0212] The processor 700 may include a controller 714 that may control inputs to the processor 700 from other processors and / or components included in the system and / or outputs from the processor 700 to other processors and / or components included in the system. The controller 714 may control data paths in the ALU 704, FPLU 706, and / or DSPU 708. The controller 714 may be implemented as one or more state machines, data paths, and / or dedicated control logic units. The gates of the controller 714 may be implemented as independent gates, FPGAs, ASICs, or any other suitable technology.

[0213] Registers 712 and cache 710 may communicate with controller 714 and core 702 via internal connections 720A, 720B, 720C, and 720D. The internal connections may be implemented as a bus, multiplexers, crossbar switches, and / or any other suitable connection technology.

[0214] Input and output to the processor 700 may be provided via a bus 716, which may include one or more conductors. The bus 716 may be communicatively coupled to one or more components of the processor 700, such as the controller 714, cache 710, and / or registers 712. The bus 716 may be coupled to one or more components of the system.

[0215] The bus 716 may be coupled to one or more external memories. The external memory may include a read-only memory (ROM) 732. The ROM 732 may be a masked ROM, an electronically programmable read-only memory (EPROM), or any other suitable technology. The external memory may include a random access memory (RAM) 733. The RAM 733 may be a static RAM, a backup static RAM, a dynamic RAM (DRAM), or any other suitable technology. The external memory may include an electrically erasable programmable read-only memory (EEPROM) 735. The external memory may include a flash memory 734. The external memory may include a magnetic storage device, such as a disk 736. In some embodiments, the external memory may be included in the system.

[0216] The present invention can be realized in any suitable form, including hardware, software, firmware or any combination of these. The present invention can be realized optionally at least in part as computer software running on one or more data processors and / or digital signal processors. The elements and components of embodiments of the present invention can be realized physically, functionally and logically in any suitable manner. In fact, the function can be realized in a single unit, in multiple units or as a part of other functional units. Like this, the present invention can be realized in a single unit, or can be physically and functionally distributed between different units, circuits and processors.

[0217] Although the present invention has been described in conjunction with some embodiments, it is not intended to be limited to the specific forms set forth herein. On the contrary, the scope of the present invention is limited only by the appended claims. In addition, although features may appear to be described in conjunction with specific embodiments, those skilled in the art will recognize that the various features of the described embodiments may be combined according to the present invention. In the claims, the term includes does not exclude the presence of other elements or steps.

[0218] In addition, although listed separately, multiple units, elements, circuits or method steps may be implemented by, for example, a single circuit, unit or processor. In addition, although individual features may be included in different claims, these features may be advantageously combined, and being included in different claims does not imply that the combination of features is not feasible and / or advantageous. In addition, including a feature in a claim category does not imply a restriction on the category, but rather indicates that the feature is equally applicable to other claim categories. In addition, the order of the features in the claims does not imply any specific order in which the features must work, and in particular the order of the individual steps in the method claims does not imply that the steps must be performed in this order. On the contrary, the steps may be performed in any suitable order. In addition, singular references do not exclude multiples. Therefore, references to "a", "an", "first", "second", etc. do not exclude multiples. The figure marks in the claims are provided only as clarification examples and should not be interpreted as limiting the scope of the claims in any way.

Claims

1. An audio device for generating a multi-channel audio signal, the device comprising: A receiver (101) is used to receive an audio data signal, wherein the audio data signal comprises: Downmixing audio signals for multi-channel signals; upmix parameterized data for upmixing the downmix audio signal; A sub-band generator (103) for generating a set of frequency sub-band signals for a sub-band of the downmix audio signal; an artificial neural network arrangement (107, 401) comprising a plurality of subband artificial neural networks (107, 401), each subband artificial neural network of the plurality of subband artificial neural networks being arranged to generate subband samples for a subband of a frequency subband representation of the multi-channel audio signal, a parameter generator (105) arranged to generate, from the upmix parameterization data, a set of upmix parameter values ​​for subbands of the frequency subband representation of the multi-channel audio signal; a generator (109) for generating the multi-channel audio signal from the sub-band samples of the sub-bands of the multi-channel audio signal; and wherein Each subband artificial neural network comprises a set of nodes arranged to receive a set of upmix parameter values ​​and samples of at least one frequency subband signal of the set of frequency subband signals for a subband for which the subband artificial neural network generates subband samples of the multi-channel audio signal.

2. The audio device according to claim 1, wherein At least a first subband artificial neural network of the plurality of subband artificial neural networks (107, 401) comprises a node for receiving parameter values ​​of an upmix parameter set for subbands other than the subband in the subband artificial neural network.

3. An audio device according to any preceding claim, wherein At least some parameters in the sets of upmix parameter values ​​for different sub-band artificial neural networks (107, 401) are identical.

4. An audio device according to any preceding claim, wherein At least some parameters in the sets of upmix parameter values ​​for different sub-band artificial neural networks (107, 401) are different.

5. An audio device according to any preceding claim, wherein The plurality of subband artificial neural networks (107, 401) for at least one subband comprises separate artificial neural networks for different channels of the multi-channel audio signal.

6. An audio device according to any preceding claim, wherein The parameter generator (105) is arranged to change the resolution of the upmix parameter set relative to the resolution of the upmix parameterization data to match the resolution of processing of the plurality of subband artificial neural networks; the resolution of the processing of the plurality of subband artificial neural networks being one of a frequency resolution for the subbands of the plurality of subband networks and a time resolution for a processing time interval.

7. The audio device according to claim 6, wherein: The parameter generator (105) comprises at least one artificial neural network having nodes for receiving parameter values ​​of the upmix parameterization data and an output node for providing a set of upmix parameter values ​​to a first subband artificial neural network of the plurality of subband artificial neural networks (107, 401).

8. An audio device according to any preceding claim, wherein The plurality of subband artificial neural networks (107, 401) can include, at least for a first subband, at least two subband artificial neural networks, the at least two subband artificial neural networks generating samples of different components of a subband signal for the first subband.

9. An audio device according to any preceding claim, wherein: The plurality of sub-band artificial neural networks (107, 401) are trained by training data having a training input audio signal comprising samples of an input multi-channel audio signal and using a cost function, the cost function comprising a component indicative of a difference between the training input audio signal and a multi-channel audio signal generated by the sub-band artificial neural network (107, 401).

10. An audio device according to any preceding claim, wherein The plurality of sub-band artificial neural networks (107, 401) are trained by having training data of a training input audio signal comprising samples of an input multi-channel audio signal and using a cost function, the cost function comprising a component indicating a difference between upmix parameters for the input audio signal and upmix parameters for the multi-channel audio signal generated by the sub-band artificial neural network (107, 401).

11. An audio device according to any preceding claim, wherein At least one sub-band artificial neural network among the plurality of sub-band artificial neural networks (107, 401) comprises: a first sub-artificial neural network having nodes receiving samples of a frequency subband signal for said subband of said subband artificial neural network and output nodes providing samples of a modified downmix audio signal; a second sub-artificial neural network having nodes receiving samples of the frequency sub-band signal for said sub-band of said sub-band artificial neural network and output nodes providing samples of the auxiliary audio signal; a third sub-artificial neural network having nodes receiving samples of the modified downmix audio signal, nodes receiving samples of the auxiliary audio signal, and nodes receiving a set of upmix parameter values ​​for the subband of the subband artificial neural network, the third sub-artificial neural network being further arranged to generate the subband samples for the subband of the frequency subband representation of the multi-channel audio signal.

12. An audio device according to any preceding claim, wherein The upmix parameter sets have different numbers of parameters for at least two subbands.

13. An audio device according to any preceding claim, wherein The upmix parameterization data provides parameterization data for sequential time intervals, and wherein at least a first subband artificial neural network of the plurality of subband artificial neural networks (107, 401) comprises a node for receiving parameter values ​​from a set of upmix parameter values ​​for another time interval in the sequential time intervals that is different from the time interval for which the subband samples of the multi-channel audio signal are generated.

14. An apparatus according to any preceding claim, wherein: The plurality of sub-band artificial neural networks (107, 401) are arranged to receive no further input data than sub-band samples of the downmix audio signal and parameter values ​​generated from the upmix parameterization data.

15. A method for generating a multi-channel audio signal, the method comprising: Receive an audio data signal, the audio data signal comprising: Downmixing audio signals for multi-channel signals; upmix parameterized data for upmixing the downmix audio signal; generating a set of frequency sub-band signals for a sub-band of the downmixed audio signal; Each subband artificial neural network of a plurality of subband artificial neural networks (107, 401) generates subband samples for a subband of a frequency subband representation of the multi-channel audio signal, generating, from the upmix parameterization data, a set of upmix parameter values ​​for subbands of the frequency subband representation of the multi-channel audio signal; The multi-channel audio signal is generated from the sub-band samples of the sub-bands of the multi-channel audio signal; and wherein Each subband artificial neural network comprises a set of nodes receiving a set of upmix parameter values ​​and samples of at least one frequency subband signal of the set of frequency subband signals for a subband for which the subband artificial neural network generates subband samples of the multi-channel audio signal.

16. A computer program product comprising computer program code means adapted to perform all the steps of the claims when said program is run on a computer.

Images