Generation of multi-channel audio signal

Abstract

An audio apparatus comprises a receiver (101) arranged to receive a downmix audio signal for a multi-channel audio signal and upmix parameter data for upmixing the downmix audio signal. A first artificial neural network (107) generates a set of feature values for the downmix audio signal from samples of the downmix audio signal. A second artificial neural network (109) has input nodes that receive second samples of the downmix audio signal and nodes that receive feature values from the set of feature values. Based on these inputs, the second artificial neural network (109) generates samples for an auxiliary audio signal for the downmix audio signal. A generator (105) generates a multi-channel audio signal from the downmix signal and the auxiliary audio signal as a function of the upmix parameter data. In many embodiments, operation may be based on sub-bands, with separate artificial neural networks for different sub-bands.

Description

Technical Field

[0001] The present invention relates to the generation of a multi-channel audio signal and in particular, but not exclusively, to the generation of a stereo signal from upmixing a mono downmix signal using upmix parameter data. Background Art

[0002] Spatial audio applications have become numerous and widespread, and increasingly form at least a part of many audiovisual experiences. In fact, new and improved spatial experiences and applications are constantly being developed, which leads to increased demands on audio processing and rendering.

[0003] For example, in recent years, virtual reality (VR) and augmented reality (AR) have received increasing attention, and many implementations and applications are entering the consumer market. In fact, equipment is being developed for presenting experiences and capturing or recording suitable data for such applications. For example, relatively low-cost equipment is being developed to allow game consoles to provide a complete VR experience. It is expected that this trend will continue, and in fact, as the market for VR and AR reaches a considerable scale in a short period of time, this trend will also accelerate. In the field of audio, a prominent area is to explore the reproduction and synthesis of realistic and natural spatial audio. The ideal goal is to produce a natural audio source so that the user cannot recognize the difference between the synthesized audio and the original audio.

[0004] A great deal of research and development work has been focused on providing efficient and high quality audio encoding and audio decoding for spatial audio. One frequently used spatial audio representation is a multi-channel audio representation, including a stereo representation, and efficient encoding of such multi-channel audio has been developed based on downmixing a multi-channel audio signal to a downmix channel having fewer channels. One of the major advances in low bitrate audio coding has been the use of parametric multi-channel coding, where the downmix signal is generated together with parametric data that can be used to upmix the downmix signal to recreate the multi-channel audio signal.

[0005] Specifically, unlike conventional mid-side or intensity coding, in parametric multi-channel audio coding, a multi-channel input signal is downmixed to a smaller number of channels (e.g., two to one) and multi-channel image (stereo) parameters are extracted. The downmixed signal is then encoded using a more conventional audio encoder (e.g., a mono audio encoder). The downmixed bitstream is multiplexed with the encoded multi-channel image parameter bitstream. The bitstream is then transmitted to a decoder where the process is reversed. The downmixed audio signal is first decoded, after which the multi-channel audio signal is reconstructed based on the encoded multi-channel image upmix parameters.

[0006] An example of stereo coding is described in the following article: E. Schuijers, W. Oomen, B. den Brinker, J. Breebaart, "Advances in Parametric Coding for High-Quality Audio", 114th AES Convention, Amsterdam, The Netherlands, 2003, Preprint 5852. In the described method, the downmixed mono signal is parameterized by exploiting the natural separation of the signal into three components (objects): transients, sinusoids and noise. More details are provided in the following article, describing how to achieve parametric stereo with low (decoder) complexity when combining parametric stereo with spectral band replication (SBR): E. Schuijers, J. Breebaart, H. Purnhagen, J. "Low Complexity Parametric Stereo Coding",116th AES,Berlin,Germany,2004,Preprint 6073.

[0007] In the described method, decoding is based on the use of a so-called decorrelation process. The decorrelation process generates a decorrelated auxiliary signal from the mono signal. In a stereo reconstruction process, the mono signal and the decorrelated auxiliary signal are used to generate an upmixed stereo signal based on the upmix parameters. Specifically, the two signals can be multiplied by a time- and frequency-dependent 2x2 matrix having coefficients determined according to the upmix parameters to provide an output stereo signal.

[0008] However, although parametric stereo (PS) and similar downmix encoding / decoding methods are a leap forward compared to traditional stereo and multi-channel encoding, the method is not optimal in all scenarios. Specifically, known encoding and decoding methods tend to introduce some distortion, changes, artifacts, etc., which may cause differences between the (original) multi-channel audio signal input of the encoder and the multi-channel audio signal recreated at the decoder. Typically, audio quality may degrade and imperfect reproduction of multiple channels may occur. In addition, the data rate may still be higher than the expected value and / or the complexity / resource utilization of the processing may be higher than expected.

[0009] Therefore, improved methods would be advantageous. Specifically, methods that allow for increased flexibility, improved adaptability, improved performance, improved audio quality, improved audio quality versus data rate tradeoff, reduced complexity and / or resource usage, reduced computational load, ease of implementation, and / or improved spatial audio experience would be advantageous. Summary of the invention

[0010] Accordingly, the Invention seeks to preferably mitigate, alleviate or eliminate one or more of the above mentioned disadvantages singly or in any combination.

[0011] According to an aspect of the present invention, there is provided an apparatus for generating a multi-channel audio signal, the apparatus comprising: a receiver arranged to receive a downmix audio signal for the multi-channel audio signal and upmix parameter data for upmixing the downmix audio signal; a first artificial neural network arranged to generate a set of feature values ​​for the downmix audio signal, the first artificial neural network having an input node for receiving a first sample of the downmix audio signal and an output node for providing the set of feature values; a second artificial neural network having an input node for receiving a second sample of the downmix audio signal and an output node arranged to provide a sample of an auxiliary audio signal for the downmix audio signal, the second artificial neural network further comprising a node for receiving feature values ​​from the set of feature values; and a generator arranged to generate the multi-channel audio signal from the downmix signal and the auxiliary audio signal according to the upmix parameter data.

[0012] The method may provide an improved audio experience in many embodiments. For many signals and scenarios, the method may provide improved generation / reconstruction of a multi-channel audio signal, as well as improved perceived audio quality. The method may provide a particularly advantageous arrangement that may allow, in many embodiments and scenarios, to facilitate and / or improve the possibility of utilizing artificial neural networks in audio processing, typically including audio encoding and / or decoding. The method may advantageously utilize (one or more) artificial neural networks to generate a multi-channel audio signal from a downmix audio signal.

[0013] The method may provide an efficient implementation and in many embodiments may reduce complexity and / or resource usage.In many cases, the method may allow the use of a downmix signal to reduce the data rate of data representing a multi-channel audio signal.

[0014] The first sample and the second sample may be the same sample, or may be different samples (or may be partially the same samples). The first sample and the second sample may be time domain samples, may be frequency domain samples, or may span a specific time and frequency range (specifically, sub-band domain samples). The samples of the auxiliary audio signal may be time domain samples, may be frequency domain samples, or may span a specific time and frequency range (specifically, sub-band domain samples).

[0015] The upmix parameter data may include parameters (values) that relate the properties of the downmix signal to the properties of the multichannel audio signal. The upmix parameter data may include data indicating the relative properties between the channels of the multichannel audio data. The upmix parameter data may include data indicating the difference in properties between the channels of the multichannel audio data. The upmix parameter data may include perceptually relevant data for the synthesis of the multichannel audio signal. These properties may be, for example, differences in phase and / or intensity and / or timing and / or correlation. In some embodiments and scenarios, the upmix parameter data may represent abstract properties that cannot be directly understood by humans / experts (but can generally promote better reconstruction / lower data rates, etc.). The upmix parameter data may include the following data, which includes at least one of the following: inter-channel intensity differences, inter-channel time differences, inter-channel correlations, and / or inter-channel phase differences for the channels of the multichannel audio signal.

[0016] The first and second artificial neural networks are trained artificial neural networks.

[0017] The first and / or second artificial neural network may be (one or more) trained artificial neural networks trained by training data, the training data comprising a training downmix audio signal and training upmix parameter data generated from the training multi-channel audio signal; the training employing a cost function to compare the training multi-channel audio signal with an upmix multi-channel signal generated from the training downmix signal and the generated auxiliary audio signal using the training upmix parameter data. The first and / or second artificial neural network may be (one or more) trained artificial neural networks trained by training data, the training data comprising training data representing a range of relevant audio sources, the relevant audio sources comprising recordings of videos, movies, telecommunications, etc.

[0018] The first and / or second artificial neural network may be (one or more) trained artificial neural networks trained by training data having training input data comprising a training downmix audio signal of a training multi-channel audio signal, and using a cost function comprising a contribution indicating a difference between a training auxiliary audio signal generated by the second artificial neural network in response to the training data and a training residual signal for the training downmix audio signal.

[0019] The generator may be arranged to generate the multi-channel audio signal by applying a matrix multiplication to the downmix signal and the auxiliary audio signal, wherein coefficients of the matrix are determined according to parameters of the upmix parameter data. The matrix is ​​time and frequency dependent.

[0020] The audio device may specifically be an audio decoding device.

[0021] According to an optional feature of the invention, the apparatus comprises a first filter bank for generating a frequency subband representation of the downmix audio signal; and wherein at least some of the second samples of the downmix audio signal are subband samples of the frequency subband representation.

[0022] Sub-band processing may provide particularly advantageous operation in many embodiments.The arrangement may be particularly suitable for sub-band processing, which may allow for reduced complexity and / or generation of an improved multi-channel audio signal.

[0023] According to an optional feature of the invention, the second artificial neural network is an artificial neural network of a first plurality of subband artificial neural networks, each subband artificial neural network of the first plurality of subband artificial neural networks being arranged to generate subband samples for a subband subset of a frequency subband representation of the auxiliary audio signal.

[0024] A particular advantage of the approach is that it can allow highly efficient sub-band processing, thereby allowing the required processing to be divided into multiple smaller artificial neural networks. This can often reduce complexity and / or generate improved multi-channel audio signals.

[0025] In many embodiments, each (or at least some) of the sub-band neural network(s) are arranged to generate sub-band samples for one sub-band of the frequency sub-band representation of the auxiliary audio signal.

[0026] According to an optional feature of the invention, the plurality of sub-band artificial neural networks comprises an artificial neural network for each sub-band in the frequency sub-band representation of the auxiliary audio signal.

[0027] This may provide highly advantageous and efficient implementation and / or operation and / or performance in many embodiments and scenarios.

[0028] According to an optional feature of the invention, the generator is arranged to generate a frequency subband representation of the multi-channel audio signal by applying a subband matrix operation to the frequency subband representation of the auxiliary audio signal and the frequency subband representation of the downmix audio signal, and is arranged to transform the frequency subband representation of the multi-channel audio signal into a time domain representation of the multi-channel audio signal.

[0029] This may provide highly advantageous and efficient implementation and / or operation and / or performance in many embodiments and scenarios.

[0030] According to an optional feature of the invention, the set of feature values ​​generated by the sub-band artificial neural network of the first plurality of sub-band artificial neural networks is common to a plurality of sub-bands of the frequency sub-band representation of the downmix audio signal.

[0031] This may provide a particularly efficient implementation and / or improved performance.

[0032] The set of feature values ​​generated by the subband neural network and common to multiple subbands can be input to multiple subband artificial neural networks that generate the multi-channel audio signal (and specifically, the second artificial neural network can be one of multiple artificial neural networks having nodes that receive a common set of feature values).

[0033] According to an optional feature of the invention, the number of input nodes of the artificial neural networks in the first plurality of sub-band artificial neural networks decreases monotonically with increasing frequency.

[0034] This may provide a particularly efficient implementation and / or improved performance.

[0035] According to an optional feature of the invention, the apparatus comprises a second filter bank for generating a frequency subband representation of the downmix audio signal; and wherein at least some of the first samples of the downmix audio signal are subband samples of the frequency subband representation.

[0036] This may provide a particularly efficient implementation and / or improved performance.

[0037] Sub-band processing may provide particularly advantageous operation in many embodiments.The arrangement may be particularly suitable for sub-band processing, which may allow for reduced complexity and / or generation of an improved multi-channel audio signal.

[0038] The first filter bank and the second filter bank may be the same or different. The subband representation of the downmix audio signal generated by the first filter bank and fed to the first plurality of subbands may use the same subbands as the subband representation of the downmix audio signal generated by the second filter bank and fed to the second plurality of subbands, or may be different in some embodiments.

[0039] According to an optional feature of the invention, the first artificial neural network is an artificial neural network of a second plurality of sub-band artificial neural networks, each sub-band artificial neural network of the second plurality of sub-band artificial neural networks being arranged to generate sub-band samples for a subset of the artificial neural networks of the first plurality of artificial neural networks.

[0040] A particular advantage of the method is that it can allow highly efficient sub-band processing, thereby allowing the required processing to be divided into multiple (optionally smaller) artificial neural networks. This can generally reduce complexity and / or generate improved multi-channel audio signals.

[0041] According to an optional feature of the invention, the sub-band samples in the second sub-band samples for at least one artificial neural network in the second plurality of artificial neural networks comprise a plurality of sub-band samples for a plurality of processing time intervals of an artificial neural network in the first plurality of artificial neural networks.

[0042] This may provide a particularly efficient implementation and / or improved performance.

[0043] According to an optional feature of the invention, the subband samples in the second subband samples for at least one artificial neural network of the second plurality of artificial neural networks include at least one subband sample of a subband for a subband representation of a downmix audio signal, for which the at least one artificial neural network does not generate subband samples for a subband representation of the auxiliary audio signal.

[0044] This may provide a particularly efficient implementation and / or improved performance.

[0045] According to an optional feature of the present invention, the first artificial neural network and the second artificial neural network are trained by a joint training process based on training data and using a cost function, wherein the training data includes: a set of samples of a downmixed audio signal generated by downmixing the training multi-channel audio signal, and a target audio signal determined based on a residual signal generated for the downmixed audio signal, and the cost function indicates the difference between an auxiliary audio signal generated for the training multi-channel audio signal and an auxiliary audio signal generated for the target audio signal.

[0046] This may provide a particularly efficient implementation and / or improved performance.

[0047] According to an optional feature of the invention, the apparatus comprises generating the eigenvalues ​​for the set of eigenvalues ​​in accordance with an analysis of the downmix audio signal.

[0048] This may provide a particularly efficient implementation and / or improved performance.

[0049] According to an aspect of the present invention, a method for generating a multi-channel audio signal is provided, the method comprising: receiving a downmix audio signal of the multi-channel audio signal and upmix parameter data for upmixing the downmix audio signal; a first artificial neural network is used to generate a set of feature values ​​for the downmix audio signal, the first artificial neural network having an input node for receiving a first sample of the downmix audio signal and an output node for providing the set of feature values; a second artificial neural network has an input node for receiving a second sample of the downmix audio signal and an output node for providing a sample of an auxiliary audio signal for the downmix audio signal, the second artificial neural network also comprising a node for receiving feature values ​​from the set of feature values; and generating the multi-channel audio signal according to the downmix signal and the auxiliary audio signal according to the upmix parameter data.

[0050] These and other aspects, features and advantages of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Embodiments of the invention are described, by way of example only, with reference to the accompanying drawings, in which:

[0052] Figure 1 illustrates some elements of an example of an audio device according to some embodiments of the present invention;

[0053] Figure 2 An example of the structure of an artificial neural network is shown;

[0054] Figure 3 An example of a node of an artificial neural network is shown;

[0055] Figure 4 illustrates some elements of an example of an audio device according to some embodiments of the present invention;

[0056] Figure 5 illustrates some elements of an example of an audio device according to some embodiments of the present invention;

[0057] Figure 6 10 shows some elements of an example of an apparatus for training an artificial neural network for an audio device according to some embodiments of the present invention; and

[0058] Figure 7 Some elements of a possible arrangement of a processor for implementing elements of an audio device according to some embodiments of the invention are illustrated. DETAILED DESCRIPTION

[0059] Figure 1 illustrates some elements of an audio device according to some embodiments of the present invention;

[0060] The audio device comprises a receiver 101, which is arranged to receive a data signal / bitstream comprising a downmix audio signal, the downmix being a downmix of a multi-channel audio signal. The following description will focus on the case where the multi-channel audio signal is a stereo signal and the downmix signal is a mono signal, but it will be appreciated that the described methods and principles are equally applicable to a multi-channel audio signal having more than two channels and a downmix signal having more than a single channel (but with fewer channels than the multi-channel audio signal).

[0061] In addition, the received data signal includes upmix parameter data for upmixing the downmix audio signal. The upmix parameter data may specifically be a set of parameters indicating the relationship between the signals (specifically stereo signals) of different audio channels of the multichannel audio signal and / or the relationship between the downmix signal of the multichannel audio signal and the audio channels. Typically, the upmix parameters may indicate time difference, phase difference, level / intensity difference and / or similarity metrics, such as correlation. Typically, the upmix parameters are provided on a per-time and per-frequency basis (time-frequency block). For example, new parameters may be provided for a group of subbands on a regular basis. The parameters may specifically include inter-channel phase difference (IPD), overall phase difference (OPD), inter-channel correlation (ICC), and channel phase difference (CPD) parameters known from parameter stereo coding (and higher channel coding).

[0062] Typically, the downmix audio signal is encoded and the receiver 101 comprises a decoder 103 which decodes the downmix audio signal (i.e. the mono signal in the specific example). It should be appreciated that if the received downmix audio signal is not encoded, the decoder 103 may not be required and may be considered as an integral part of the receiver 101.

[0063] The receiver 101 is coupled to a generator 105, which generates a multi-channel audio signal according to the downmix signal. The generator 105 is arranged to generate a multi-channel audio signal from the downmix audio signal and the auxiliary audio signal according to the parameter upmix data. For the stereo case, the generator can generate an output multi-channel audio signal specifically by applying a 2x2 matrix multiplication to the samples of the downmix audio signal and the auxiliary audio signal. The coefficients of the 2x2 matrix are determined by the upmix parameters of the upmix parameter data, usually on the basis of time and frequency band. For other upmix operations, such as from a mono or stereo downmix signal to a five-channel multi-channel audio signal, the generator 105 can apply a matrix multiplication of appropriate dimensions.

[0064] It should be appreciated that those skilled in the art will appreciate many different methods of generating such a multi-channel audio signal based on the downmix audio signal and the auxiliary audio signal and determining suitable matrix coefficients based on the upmix parameter data, and any suitable method may be used. Specifically, those skilled in the art are familiar with various methods of upmixing of parameters based on downmix and auxiliary audio signals.

[0065] In conventional systems, upmixing involves generating an auxiliary audio signal in the form of a decorrelated signal of a monophonic audio signal. It has been found that by generating a decorrelated signal and mixing it with the monophonic audio signal, an improvement in the quality of the upmixed signal can be perceived, and decoders have therefore been developed to take advantage of this. The decorrelated signal is typically generated by a decorrelator in the form of an all-phase filter and applied to the monophonic audio signal. However, while the use of such an all-pass filter tends to result in a multi-channel audio signal that is perceived to be of improved quality, it is still not ideal and some audio quality degradation is often perceived.

[0066] Figure 1 An audio device of the invention uses a method which has been found to tend to provide improved perceived audio quality in many scenarios and for many different audio signals.

[0067] In this method, the decorrelated signal is not generated by direct filtering of the downmix / mono audio signal, but is an auxiliary audio signal generated by a specific arrangement of a trained artificial neural network, wherein the generator 105 uses the auxiliary audio signal to generate a multi-channel audio signal based on the upmix parameters.

[0068] Figure 1 The device specifically includes a first artificial neural network 107, which is arranged to receive a sample of a downmix audio signal and feed it to an input node of the first artificial neural network. The output node of the first artificial neural network 107 provides a set of eigenvalues ​​for the downmix audio signal. The set of eigenvalues ​​may include multiple values ​​(such as specific scalar values) reflecting the attributes or characteristics of the downmix audio signal. Typically, the first artificial neural network 107 may include much more input nodes than output nodes, and therefore a relatively small number of eigenvalues ​​may be generated based on a relatively large number of samples. Therefore, the eigenvalues ​​can provide a highly compressed and simplified representation of some attributes or characteristics of the downmix audio signal. For example, the first artificial neural network 107 may have 1028 input nodes and 16 output nodes, thereby providing a highly compressed set of values, which depends on the attributes of the downmix audio signal.

[0069] The apparatus further comprises a second artificial neural network 109 coupled to the first neural network and the decoder 103 / receiver 105. The second artificial neural network 109 specifically comprises an input node for receiving a sample of the downmix audio signal. In addition, the second artificial neural network 109 comprises a node receiving a contribution of an eigenvalue from the set of eigenvalues ​​generated by the first artificial neural network 107. Such a node may be an input node of an input layer, which also comprises a node receiving a sample of the downmix audio signal, or one, more or all of the nodes receiving the eigenvalues ​​may be nodes of a layer different from the input layer of the downmix audio signal. For example, some or all of the nodes receiving contributions from the eigenvalues ​​may be part of a hidden layer or a processing layer of the second artificial neural network 109.

[0070] The second artificial neural network 109 has output nodes providing samples of the auxiliary audio signal, which may then be fed to the generator 105 where the upmixing operation is done.

[0071] Thus, in this method, the upmixing process is not based on applying a decorrelation filter to the downmix audio signal to generate a decorrelated signal, which is then combined with the downmix audio signal to generate a multi-channel audio signal. Instead, the trained artificial neural network structure generates an auxiliary audio signal, which can specifically replace the decorrelated signal used in the conventional upmix decoder. The first artificial neural network 107 and the second artificial neural network 109 both have input nodes for receiving samples of the downmix audio signal, but in addition, the output of the first artificial neural network 107 is introduced into the second artificial neural network 109 and is therefore used to control and adjust the processing of the second artificial neural network 109. Therefore, even if the weights and coefficients are constant, the second artificial neural network 109 is not just a fixed filter or trained network, but can be seen as an adaptive or variable operation that is adjusted based on the results of the first artificial neural network 107.

[0072] As will be described in detail later, different methods can be used to train the neural network, and in particular, the overall training can seek to obtain the output of the audio device as a multi-channel audio signal that corresponds most closely to the original multi-channel audio signal. Therefore, the device can be trained to provide an auxiliary audio signal, thereby most effectively achieving accurate reconstruction of the multi-channel audio signal. Compared with traditional methods, such a signal is not necessarily a decorrelated signal, but the second artificial neural network 109 will be trained to obtain an auxiliary audio signal that is most suitable for combining with the downmix audio signal to generate a multi-channel audio signal. Such a signal is usually not a decorrelated signal of the downmix audio signal, but may be, for example, a partially correlated signal, and in fact in many cases may be closer to the actual residual signal produced by the original downmix of the multi-channel audio signal. Therefore, the user of the trained artificial neural network can allow the decoder to inherently and automatically consider and compensate for the effects that may be introduced at the encoder end.

[0073] Similarly, the first artificial neural network 107 and the generated features are not specific feature values ​​representing specific attributes or characteristics of the signal that are significant to humans. Instead, the first artificial neural network 107 can be trained so that it is automatically adapted to provide feature values ​​that are particularly suitable for adapting the second artificial neural network 109, thereby providing improved output values ​​to accurately reproduce the multi-channel audio signal.

[0074] In the method, the artificial neural network device is therefore arranged to generate a second auxiliary "helper" signal to assist and improve the multi-channel reconstruction. For example, for a stereo signal, the encoder can generate a downmix signal as c*(l+r), where l and r represent the left channel signal and the right channel signal respectively, and c is a time- and frequency-dependent scaling factor. The corresponding second signal for the ideal reconstruction can be shown as d*(lr), where d is again related to time and frequency. The two signals are not necessarily completely decorrelated, and a major advantage of the described method is that the artificial neural network arrangement will generate an auxiliary audio signal that tends to reach the ideal signal d*(lr) compared to simply trying to decorrelate the mono mix. This can generally provide a significantly improved reconstruction of the original multi-channel audio signal.

[0075] The generation of the auxiliary audio signal is further improved by the second artificial neural network 109 generating a signal that is adjusted based on the set of feature values ​​generated by the first artificial neural network 107. This adjustment may significantly improve the reconstruction relative to the case where no adjustment is included.

[0076] Furthermore, the particular arrangement described allows for a very efficient implementation with relatively low complexity and resource usage.

[0077] The artificial neural network used in the described function may be a network of nodes arranged in layers, and each node holds a node value. Figure 2 An example of a portion of an artificial neural network is illustrated.

[0078] The node value for a given node can be calculated to include contributions from some or usually all nodes in the previous layer of the artificial neural network. Specifically, the node value for a node can be calculated as the weighted sum of the node values ​​output by all nodes in the previous layer of the node. Typically, a bias can be added and then the result is processed by an activation function. The activation function typically provides an important part for each neuron by providing nonlinearity. This nonlinearity and activation function play a significant role in the learning and adaptation process of the neural network. Therefore, the node value is generated based on the node value of the previous layer.

[0079] The artificial neural network may specifically include an input layer 201, which includes a plurality of nodes receiving input data values ​​for the artificial neural network. Therefore, the node values ​​of the nodes of the input layer are usually directly the input data values ​​of the artificial neural network, and therefore may not be calculated based on other node values.

[0080] The artificial neural network may also include zero, one or more hidden layers 203 or processing layers. For each of such layers, the node value is usually generated based on the node value of the previous layer node, and specifically a weighted combination and added bias, followed by an activation function (such as sigmoid, ReLU or tanh function can be applied).

[0081] Specifically, if Figure 3 As shown in , each node (also called a neuron) can receive input values ​​(from the nodes in the previous layer) and calculate the node value based on these values. Typically, this involves first generating a value as a linear combination of the input values, where each input value is weighted by a weight:

[0082]

[0083] Among them, w represents the weight, x represents the node of the previous layer, and n refers to the index of different nodes in the previous layer.

[0084] An activation function can then be applied to the resulting combination. For example, the node value l can be determined as:

[0085] l=f(k)

[0086] Wherein, the function may be, for example, a rectified linear unit (described in Xavier Glorot, Antoine Bordes, Yoshua Bengio Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, PMLR 15:315-323, 2011) function:

[0087] f(k)=ReLU(k)=max(0,k)

[0088] Other commonly used functions include sigmoid-type functions or tanh functions. In many embodiments, multiple functions can be used to calculate node outputs or values. For example, an activation function can be used to combine ReLU and Sigmoid functions, such as:

[0089] f(k)=ReLU(k)+σ(k)

[0090] Such operations can be performed by every node of the artificial neural network (usually except the input node).

[0091] The artificial neural network also includes an output layer 205, which provides the output from the artificial neural network, i.e., the output data of the artificial neural network are the node values ​​of the output layer. For hidden layers / processing layers, the output node values ​​are generated by a function of the node values ​​of the previous layer. However, unlike hidden layers / processing layers (where the node values ​​are usually not accessible or further used), the node values ​​of the output layer are accessible and provide the results of the artificial neural network operation.

[0092] Many different network structures and toolboxes have been developed for artificial neural networks, and in many embodiments, artificial neural networks can be based on such networks. An example of a network architecture suitable for the above application is the WaveNet of van den Oord et al., which is described in the following article: Oord, Aaron van den, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. "Wavenet: A generative model for raw audio." arXiv preprint arXiv:1609.03499 (2016).

[0093] WaveNet is an architecture for synthesizing time-domain signals using dilated causal convolutions and has been successfully applied to audio signals. For WaveNet, the following activation functions are usually used:

[0094] z=tanh(W f，k *x)⊙σ(W g，k *x),

[0095] In which * represents the convolution operator, ⊙ represents the element-wise multiplication operator, σ(·) is the sigmoid function, k is the layer index, f and g represent the filter and gate, respectively, and W represents the learned weight of the artificial neural network. The filtered product of the formula can generally provide a filtering effect, while the gated product provides the weight of the result, which in many cases can effectively allow the contribution of a node to be reduced to essentially zero (i.e., it can allow or "cut off" a node that contributes to other nodes, thereby providing a "gate" function). In different cases, the gate function may result in a negligible output of the node, while in other cases, it will make a substantial contribution to the output. Such a function can substantially help neural networks learn and train effectively.

[0096] In some cases, the artificial neural network can also be arranged to include additional contributions, thereby allowing the artificial neural network to be dynamically adjusted or customized for specific desired properties or characteristics of the generated output. For example, a set of values ​​can be provided to adjust the artificial neural network. These values ​​can be included by providing contributions to some nodes of the artificial neural network. These nodes may be specific input nodes, but may generally be nodes of hidden layers or processing layers. For example, such adaptive values ​​can be weighted and added as contributions to the weighted sum / correlation value for a given node. For example, for WaveNet, such adaptive values ​​can be included in the activation function. For example, the output of the activation function can be given as:

[0097] z=tanh(W f，k *x+V f，k *y)⊙σ(W g，k *x+V g，k *y)

[0098] where y is a vector representing the adjusted values ​​and V represents the appropriate weights for these values.

[0099] The above description relates to neural network methods that may be applicable to many embodiments and implementations. However, it should be understood that many other types and structures of neural networks may be used. In fact, many different methods for generating neural networks have been and continue to be developed, including neural networks that use complex structures and processes different from those described above. The method is not limited to any particular neural network method, and any suitable method may be used without detracting from the invention.

[0100] In many embodiments, the audio device is arranged to perform sub-band processing. In particular, Figure 4 As shown in Figure 1 The device may include a filter bank arranged to generate a frequency or sub-band representation of the downmix audio signal. The filter bank may be a quadrature mirror filter (QMF) bank or may be implemented, for example, by a fast Fourier transform (FFT), but it should be understood that many other filter banks and methods for separating an audio signal into a plurality of sub-band signals are known and may be used. The filter bank may specifically be a complex-valued pseudo-QMF bank, thereby generating, for example, 32 or 64 complex-valued sub-band signals.

[0101] In many embodiments, filter bank 401 is arranged to generate a group of sub-band signals for sub-bands with the same bandwidth. In other embodiments, filter bank 401 can be arranged to generate sub-band signals with sub-bands of different bandwidths. For example, a higher frequency sub-band may have a higher bandwidth than a lower frequency sub-band. In addition, the sub-bands can be grouped together to form a higher bandwidth sub-band.

[0102] Typically, the bandwidth of a sub-band may be in the range of 10 Hz to 10,000 Hz.

[0103] In some such embodiments, the artificial neural network generating samples for the auxiliary audio signal is arranged to receive sub-band samples, i.e. samples of a sub-band audio signal. Figure 4 The apparatus comprises a plurality of sub-band neural networks 109, each neural network 109 being arranged to receive sub-band samples for a sub-band generated by a filter bank according to a downmix signal. Each sub-band artificial neural network also comprises a set of eigenvalues, and it proceeds to generate sub-band samples of a multi-channel audio signal for the sub-band.

[0104] The subband samples from the subband artificial neural network are fed to the generator 105, which proceeds to generate the reconstructed multi-channel audio signal. For example, in some embodiments where a subband representation of the multi-channel audio signal is required (e.g., because subsequent processing is also based on subbands), the generator 105 can simply output the subband samples from the subband artificial neural network, possibly according to a specific structure or format. In many embodiments, the generator 105 includes a function for converting the subband representation of the reconstructed multi-channel audio signal into a time domain representation. The generator 105 may specifically include a synthesis filter bank that performs the inverse operation of the filter bank 401, thereby converting the subband representation into a time domain representation of the multi-channel audio signal.

[0105] The generator may be specifically arranged to generate a frequency / subband domain representation of the multi-channel audio signal by processing a frequency or subband domain representation of the downmix audio signal and a frequency / subband domain representation of the auxiliary audio signal. Thus, the processing of the generator 105 may be a subband processing, such as performing a matrix multiplication in each subband on the subband samples of the downmix audio signal and the auxiliary audio signal generated by the corresponding subband artificial neural network.

[0106] The resulting subband / frequency domain representation may then be used directly or may be converted to a time domain representation using a suitable synthesis filter bank, in particular may be applied by separate synthesis filters for each channel.

[0107] Each of the plurality of sub-band neural networks may also be referred to as an auxiliary sub-band (domain) artificial neural network or more concisely as an auxiliary sub-band artificial neural network. The comments previously provided with respect to the second artificial neural network 109 will also apply to the auxiliary sub-band artificial neural network, and in fact the second artificial neural network 109 may be considered as one of the auxiliary sub-band artificial neural networks.

[0108] Hence, in this arrangement each auxiliary sub-band artificial neural network receives sub-band samples of a sub-band of the auxiliary sub-band artificial neural network, and further all auxiliary sub-band artificial neural networks are arranged to receive a set of feature values ​​from the first artificial neural network 107 .

[0109] Each auxiliary subband artificial neural network generates subband samples for a subband subset of a frequency subband representation of the auxiliary audio signal, and typically generates (only) subband samples for the subband for which it receives input samples from the filterbank 401 .

[0110] In many embodiments, the apparatus comprises an artificial neural network for each subband represented by the frequency subband of the auxiliary audio signal generated by the filter bank. Thus, in many embodiments, the output samples for each subband of the filter bank 401 are fed to an input node of an auxiliary subband artificial neural network, which then generates subband samples of the auxiliary audio signal for that subband. Thus, in many embodiments, the subband processing for each subband may be completely separated.

[0111] However, it should be understood that in some embodiments, auxiliary subband artificial neural networks may be provided for only zero, one or some subbands, while for other subbands, auxiliary subband artificial neural networks receiving samples from multiple subbands may be used, or indeed for some subbands, the auxiliary subband artificial neural network may not be applied (e.g., conventional decorrelation may be used for some subbands, such as typically for higher subbands).

[0112] Thus, in the example, the generation of the auxiliary audio signal is performed on a subband-by-subband basis, and there is a separate and individual artificial neural network in each subband. Each artificial neural network is trained to provide output samples for the subband for which they are provided with input subband samples. However, the subband artificial neural network is also adapted to be further adjusted based on the feature values ​​generated by the first artificial neural network 107.

[0113] It has been found that this method provides a very advantageous generation of auxiliary audio signals, which allows the reconstruction of multi-channel audio signals with very high quality. In addition, it can achieve efficient operation, significantly reduce complexity and / or generally significantly reduce computing resource requirements. Sub-band artificial neural networks are often significantly smaller than the single complete artificial neural network required to generate the complete signal. Usually, fewer nodes are required for processing, and the number of layers that may even be required is also less, resulting in a significant reduction in the number of operations and calculations required to implement the artificial neural network function. Although more artificial neural networks are needed to cover all sub-bands, smaller artificial neural networks usually result in a significant reduction in the number of overall operations required, and therefore reduce the overall computing resource requirements. In addition, it can allow a more efficient learning process in many cases.

[0114] Thus, the subband arrangement can provide a computationally efficient method for allowing an artificial neural network to be implemented to assist in decoding audio data comprising a downmix audio signal and upmix parameter data. The described systems and methods allow reconstruction of a high-quality multi-channel audio signal and can generally achieve significantly improved audio quality compared to conventional methods. Furthermore, a computationally efficient decoding process can be achieved. The subband and artificial neural network based methods may also be compatible with other processes that use subband processing.

[0115] In some embodiments, subband processing may be more flexible than strict subband-by-subband processing. For example, in some embodiments, each auxiliary subband artificial neural network may receive subband samples not only from the subband itself, but also from one or more other subbands. For example, in some embodiments, an auxiliary subband artificial neural network for a subband may also receive samples of a downmixed audio signal from one or two neighboring / adjacent subbands. As another example, in some embodiments, one or more auxiliary subband artificial neural networks may also receive input samples from one or more subbands that include harmonics (or subharmonics) of the subband frequency. For example, a subband with a center frequency of around 500 Hz may also receive frequencies from a subband with a center frequency of around 1000 Hz. These additional subbands that have a specific relationship to the subbands of the auxiliary subband artificial neural network may provide additional information that may allow improved auxiliary subband artificial neural networks to be generated for certain audio signals.

[0116] In some embodiments, all auxiliary sub-band artificial neural networks may have the same properties and size. Specifically, in many embodiments, all auxiliary sub-band artificial neural networks may have the same number of input nodes and output nodes, and may have the same internal structure. This approach can be used, for example, in embodiments where all sub-bands have the same bandwidth.

[0117] However, in some embodiments, the auxiliary sub-band artificial neural networks may include different neural networks. Specifically, in some embodiments, the number of input nodes for the auxiliary sub-band artificial neural networks may be different for at least two artificial neural networks. Therefore, in some embodiments, for different sub-bands and auxiliary sub-band artificial neural networks, the number of input samples included in determining output samples may be different.

[0118] In some embodiments, the number of samples / input nodes for certain lower frequency bands may be higher than the number of samples / input nodes for certain higher frequency bands. In fact, the number of samples / input nodes may decrease monotonically with increasing frequency. Therefore, the lower frequency auxiliary sub-band artificial neural network may be larger and consider more input samples than the higher frequency auxiliary sub-band artificial neural network. For example, this approach can be combined with sub-bands with different bandwidths, such as when the lower frequency sub-band may have a higher bandwidth than the higher frequency bandwidth.

[0119] In many cases, this approach can provide a better trade-off between achievable audio quality and computational complexity and resource usage. It can provide a tighter adaptation of the system to reflect the typical characteristics of the audio, resulting in more efficient processing.

[0120] In the above method, the auxiliary sub-band artificial neural network is provided with the same set of feature values. In some embodiments, the same set of feature values ​​may be provided to only some of the auxiliary sub-band artificial neural networks. However, in many embodiments, using the same set of feature values ​​may provide higher efficiency and performance. It may generally reduce the complexity and resource usage of generating the set of feature values. In addition, it may provide improved operation in many scenarios, and each auxiliary sub-band artificial neural network may consider all available information provided by the feature values, and thus improved adaptability of the auxiliary sub-band artificial neural network may be achieved.

[0121] However, in some embodiments, different artificial neural networks may be provided with different sets of feature values. For example, the first artificial neural network 107 may generate a set of feature data values, and different subsets of them may be provided to different auxiliary sub-band artificial neural networks. In other embodiments, some feature sets may also be generated to include some feature values, for example, provided manually or generated by analysis of the downmix audio signal. For example, harmonics or peaks may be detected in the downmix audio signal. For example, such data may only be applied to certain auxiliary sub-band artificial neural networks. For example, detected peaks or harmonics may only be indicated to the auxiliary sub-band artificial neural networks in the sub-band in which they are detected.

[0122] Therefore, in many embodiments, different sets of feature values ​​may be provided for different auxiliary sub-band artificial neural networks. In many cases, this may include that some feature values ​​are the same and some feature values ​​are different for different sets of auxiliary sub-band artificial neural networks.

[0123] In the above method, a single first artificial neural network 107 is used to generate feature values ​​for a set of feature values, while a plurality of auxiliary sub-band artificial neural networks process the sub-bands to downmix the audio signal.

[0124] In other embodiments, the generation of a set of eigenvalues ​​can be based on subbands. In some embodiments, a separate filter bank can be applied to the downmix audio signal to provide a set of subband signals, which are then fed to the corresponding subband artificial neural networks, each of which generates a set of eigenvalues. Such multiple artificial neural networks will also be referred to as feature set artificial neural networks. The set of features generated by the feature set artificial neural network can then be input to the auxiliary subband artificial neural network. The subbands generated by such a filter bank do not have to be the same as the subbands generated by the filter bank 401 for the auxiliary subband artificial neural network to generate subband samples. For example, each subband used to determine a feature set can include a different number of subbands for the auxiliary subband artificial neural network, and the feature set determined by a feature set artificial neural network can be fed to the appropriate auxiliary subband artificial neural network.

[0125] However, in many embodiments, the sub-bands used for feature generation may be the same as the sub-bands used for the auxiliary sub-band artificial neural network. Figure 5 An example of such a method is shown. In this example, rather than generating a set of feature values ​​by a single first artificial neural network 107, Figure 1 The apparatus in is modified to include a plurality of artificial neural networks, each artificial neural network generating a set of feature values.

[0126] The comments previously provided with respect to the first artificial neural network 107 will also apply to the feature set artificial neural network, and in fact the first artificial neural network 107 may be considered as one of the feature set artificial neural networks.

[0127] In the method, each feature set artificial neural network generates a set of eigenvalues ​​that are applicable only to a subset of auxiliary sub-band artificial neural networks. In fact, in many embodiments, each feature set artificial neural network can generate a set of eigenvalues ​​for one of the auxiliary sub-band artificial neural networks. In some embodiments, the device can include the same number of feature set artificial neural networks and auxiliary sub-band artificial neural networks, and specifically there can be one feature set artificial neural network and one auxiliary sub-band artificial neural network for each sub-band of the filter bank 401. In other embodiments, there may be different numbers of feature set artificial neural networks and auxiliary sub-band artificial neural networks. For example, a feature set artificial neural network may receive input samples for a group of sub-bands and generate a set of eigenvalues ​​for the group of sub-bands. This set of eigenvalues ​​can then be applied to the group of auxiliary sub-band artificial neural networks for these sub-bands.

[0128] This method of generating a set of eigenvalues ​​based on subbands can provide better results in many cases because it allows a more accurate set of eigenvalues ​​to be generated and used to adjust the auxiliary subband artificial neural network. It can also reduce complexity and / or resource usage in many scenarios. For example, in many embodiments, fewer values ​​in the set of eigenvalues ​​can be implemented, thereby allowing the complexity of the feature set artificial neural network and the auxiliary subband artificial neural network to be reduced. In addition, typically, the feature set artificial neural network has fewer inputs and is much smaller than an artificial neural network covering the entire bandwidth.

[0129] In many such embodiments, each feature set artificial neural network may generate a set of feature values ​​for a given subset of subbands (and typically for a single subband) based on subband samples of the subset of subbands. However, in addition, one or more feature set artificial neural networks may also include subbands from one or more other subbands, i.e., the input to the feature set artificial neural network may have input nodes that receive subband samples of subbands for which no set of feature values ​​has been generated by the feature set artificial neural network.

[0130] As a specific example, in many embodiments, each feature set artificial neural network may receive as input subsamples not only from the subband of the set of feature values ​​it generates, but also from adjacent subbands.

[0131] This approach may generally generate an improved set of eigenvalues, which may lead to improved audio quality. In particular, it has been found that taking surrounding subbands into account may allow the set of eigenvalues ​​to better reflect the temporal resolution of the downmix audio signal / multi-channel audio signal. It has been found that this approach in particular allows a better representation of temporal peaks.

[0132] In some embodiments, one or more of the feature set artificial neural networks, or the first artificial neural network 107 in the case where only one such artificial neural network is included, may further include as input sub-band samples outside the time interval of samples from which the corresponding auxiliary sub-band artificial neural network generated the auxiliary audio signal.

[0133] Specifically, the processing of the audio device is performed on a frame-by-frame basis, wherein a time interval / frame of the received downmix audio signal is processed to generate output samples of the multi-channel audio signal for the time interval / frame. Therefore, for each frame, the filter bank 401 generates sub-band samples, which are fed to a feature set artificial neural network, which generates a set of feature values ​​for the time interval, and the feature set artificial neural network generates sub-band samples of a sub-band representation of the multi-channel audio signal based on the set of sub-bands and feature values.

[0134] Thus, in particular, each feature set artificial neural network operates in blocks, wherein each operation generates a set of output samples from a set of input samples corresponding to time intervals of the downmix audio signal / multichannel audio signal that generate output samples of the multichannel audio signal.

[0135] In some embodiments, in addition to the appropriate sub-band samples generated for the current time interval, one or more feature set artificial neural networks may also receive sub-band samples for another time interval, such as typically received from one or more adjacent time intervals. For example, in some embodiments, one or more feature set artificial neural networks may also include sub-band samples for the previous and next time intervals.

[0136] In many embodiments, this approach may provide an improved set of feature values, thereby improving audio quality.

[0137] Artificial neural networks are adapted to specific purposes through a training process, which is used to adjust / tune / change the weights and other parameters (e.g., biases) of the artificial neural network. It should be appreciated that many different training processes and algorithms are known to be used for training artificial neural networks. Typically, training is based on a large training set, in which a large number of examples of input data are provided to the network. In addition, the output of the artificial neural network is typically compared (directly or indirectly) to an expected or ideal result. A cost function can be generated to reflect the expected results of the training process. In a typical scenario known as supervised learning, the cost function typically represents the distance between the prediction and the actual situation for specific input data. Based on the cost function, the weights can be changed, and by iteratively modifying the weights, the artificial neural network can be adjusted toward a state where the cost function is minimized.

[0138] In more detail, during the training step, the neural network may have two different information flows from input to output (forward pass) and from output to input (backward pass). In the forward pass, the data is processed by the neural network as described above, and in the backward pass, the weights are updated to minimize the cost function. Typically, this backward propagation follows the gradient direction of the cost function landscape. In other words, by comparing the predicted output with the actual situation for a batch of data inputs, the direction of minimization of the cost function can be estimated and propagated backward by updating the weights accordingly. Other known methods for training artificial neural networks include, for example, the Levenberg-Marquardt algorithm, the conjugate gradient method, and Newton's method.

[0139] In the present case, the training may specifically include a training set containing a possibly large number of multi-channel audio signals or corresponding downmixed audio signals. The training set may include audio signals representing a plurality of different audio sources, such as video recordings, movies, telecommunications, etc. In some embodiments, the training data may even include non-audio data, such as training in combination with training data from other sources, such as text data, etc.

[0140] In some embodiments, the training data may be a multi-channel audio signal in a time period corresponding to a processing time interval of the artificial neural network being trained, for example, the number of samples in the training multi-channel audio signal may correspond to the number corresponding to the input nodes of the (one or more) artificial neural networks being trained. Thus, each training example may correspond to one operation of the (one or more) artificial neural networks being trained. However, typically, a batch of training samples is considered at each step to speed up the training process. In addition, many upgrades of gradient descent may also speed up convergence or avoid local minima in the cost function landscape.

[0141] For each training multi-channel audio signal, the training processor may perform a downmix operation to generate a downmix audio signal and corresponding upmix parameter data. Thus, the encoding process applied to the multi-channel audio signal during normal operation may also be applied to the training multi-channel audio signal to generate downmix and upmix parameter data.

[0142] Furthermore, in some embodiments, the training processor may generate a residual signal reflecting the difference between the downmix audio signal and the multi-channel audio signal, or more typically representing a portion of the multi-channel audio signal that is not correctly represented by the downmix audio signal. For example, in many embodiments, the training processor may generate a downmix signal and may additionally generate a residual signal that, when used in upmixing based on the upmix parameter data, will result in a (more) accurate multi-channel audio signal to be reconstructed.

[0143] Specifically, for stereo multi-channel audio signals, the training processor can use a parametric stereo scheme (e.g., according to a suitable standardized method). Such encoding will apply frequency and time-related matrix operations, such as rotating the input stereo signal to generate a downmix signal and a residual signal. For example, a 2x2 matrix multiplication / complex value multiplication is usually applied to the input stereo signal, such as to basically align one of the rotated channel signals to have a maximum signal value. The channel can be used as a mono signal, and the rotation is usually performed on a frame basis. The rotation value can be stored as part of the upmix parameter data (or the parameter that allows the rotation value to be determined can be included in the upmix parameter data). Therefore, in the synthesis device, the opposite rotation can be performed to reconstruct the stereo signal. The rotation of the stereo signal will produce another stereo signal, in which one channel is therefore aligned with the maximum intensity. In order to reduce the data rate, the parametric stereo encoder will usually discard another channel. In traditional PS decoding, a decorrelated signal is usually generated at the decoder and used for the upmix process. In current training methods, this second signal may be used as a residual signal for the downmix, since it may represent information that was discarded in the encoder and therefore it represents the ideal signal to be reconstructed in the decoder as part of the upmix process.

[0144] Thus, in some embodiments, the training processor may generate a training downmix signal and / or a training residual signal from the training multichannel audio signal. The training downmix signal may be fed to an arrangement of the first artificial neural network 107 and the second artificial neural network 109, or equivalently to an arrangement of a feature set artificial neural network and an auxiliary subband artificial neural network, i.e. samples of the training downmix audio signal are fed to the neural network using the same processing (e.g. including subband filtering etc.) as applied by the audio device to the downmix audio signal during normal operation.

[0145] The output of the neural network operation is then determined and a cost function is applied to determine a cost value for each training downmix audio signal and / or a combined set of training downmix audio signals (eg, determine an average cost value for the training set). The cost function may include various components.

[0146] Typically, the cost function will include at least one component reflecting how close the generated signal is to a reference signal, the so-called reconstruction error. In some embodiments, the cost function will include at least one component reflecting how close the generated signal is to a reference signal from a perceptual perspective.

[0147] For example, in some embodiments, the auxiliary audio signal generated by the second artificial neural network 109 for a given training downmix audio signal / multi-channel audio signal can be compared with the residual signal of the training downmix audio signal / multi-channel audio signal. A cost function contribution / combination can be generated that reflects the difference between the generated auxiliary audio signal and the reference residual signal. This process can be generated for all training sets to generate an overall cost function.

[0148] Figure 6 Such an example is illustrated. In this example, the downmixer 601 can receive a training multi-channel audio signal, which is a stereo signal in a specific example. For a given training signal, the downmixer 601 performs a downmix operation to generate a training downmix signal, which is a training mono audio signal and a residual signal in a specific example. The downmix audio signal is fed to a preprocessor 603, which can specifically generate downmix audio signal samples to input into an artificial neural network. The preprocessor 603 specifically performs the same operations performed in the synthesis device to generate samples for the artificial neural network, and in fact the same functions can usually be used, i.e., the decoder functions are also used in the training process to generate artificial neural network input samples. In addition, the preprocessor 603 may include functions corresponding to the encoder / decoding process of the training downmix audio signal, such as including quantization, etc.

[0149] The output of the preprocessor 603 is Figure 1 The first artificial neural network 107 and the second artificial neural network 109 are fed to the first artificial neural network 107 and the second artificial neural network 109 in the same manner as the audio device of the training downmix audio signal. The first artificial neural network 107 and the second artificial neural network 109 are coupled to each other so that the output set of feature values ​​generated by the first artificial neural network 107 is fed to the second artificial neural network 109. Therefore, the output of the second artificial neural network 109 corresponds to the sample of the auxiliary audio signal generated for the training downmix audio signal.

[0150] Therefore, for a given training multi-channel audio signal, Figure 6 The training system performs Figure 1The audio device of the artificial neural network performs the same operation as that of the audio device, thereby generating an auxiliary audio signal that the audio device would generate if the artificial neural network had the same data / configuration (coefficients, biases, etc.).

[0151] The output of the second artificial neural network 109 is fed to a comparator 605, which proceeds to compare the generated auxiliary audio signal with the residual signal generated by the downmixer 601. As described above, the residual signal can in principle achieve a substantially perfect reconstruction of the multi-channel audio signal, and can therefore be considered to be an approximation of the ideal auxiliary audio signal. Therefore, the comparison between the generated auxiliary audio signal and the residual signal provides an indication of how favorable the generated auxiliary audio signal is. A cost value can therefore be determined based on the comparison, and in particular, the greater the difference, the higher the cost value.

[0152] It should be appreciated that many different methods can be used to determine a cost value reflecting the difference between the signals. For example, a correlation can be performed where the cost value has a monotonically decreasing value as the correlation value increases. As another example, two signals can be subtracted from each other and a power measure of the difference signal can be used as the cost value. It should be appreciated that many other methods are available and can be used.

[0153] Thus, in an example, the cost function generates a cost value that reflects how well the generated auxiliary audio signal matches the corresponding residual signal of the training multi-channel audio signal.

[0154] Based on the cost value, the training processor 607 can adjust the weights of the artificial neural network. For example, a back propagation method can be used. Specifically, the training processor 607 can adjust the weights of the first artificial neural network 107 and the second artificial neural network 109 based on the cost value. For example, given the derivative (representing the slope) of the weight with respect to the cost function, the weight value will be modified to be in the direction of the slope. For simple / minimum considerations, reference can be made to the training of a perceptron (a single neuron) in the case of a single data input being passed backward.

[0155] This process can be iterated until the artificial neural network is considered trained. For example, a predetermined number of iterative trainings can be performed. As another example, training can continue until the weight change is less than a predetermined amount. It is also common to implement a validation stop, where the network is tested again against a validation metric and stops when the expected results are achieved.

[0156] As a specific example, a stereo signal can be fed to a conventional PS downmix module, which generates both a downmix signal and an ideal residual signal, i.e., a residual signal that will allow (close to) perfect reconstruction of the waveform at the decoder end. Using pairs of mono signals and residual signals, a first artificial neural network 107 generates a set of feature values ​​that can represent the audio signal (frame). This set of feature values ​​is fed together with the mono audio signal to a second artificial neural network 109 that generates an auxiliary audio signal. When the artificial neural network is deep enough, this arrangement can be trained using a cost function such as RMSE (root mean square error of the auxiliary audio signal relative to the residual signal), resulting in the artificial neural network arrangement learning what the auxiliary audio signal should be for a given mono audio signal.

[0157] In this arrangement, the first artificial neural network 107 and the second artificial neural network 109 are respectively jointly trained using the same downmix audio signal and cost function. The weights of the first artificial neural network 107 and the second artificial neural network 109 are both updated based on the same training data and the same cost function and downmix audio signal.

[0158] In many embodiments, the residual signal can be used directly when comparing with the generated auxiliary audio signal. However, more generally, a target audio signal generated from the residual audio signal can be generated and used to compare with the generated auxiliary audio signal. The target audio signal can be generated by applying a function or signal processing application to the residual signal. For example, a scaling / level setting of the residual audio signal can be applied to generate the target audio signal. For another example, a filtering operation can be applied to the residual audio signal to generate the target audio signal. As another example, scaling can be applied to the residual signal to minimize the difference and / or maximize the correlation with the generated auxiliary audio signal.

[0159] In some embodiments, the cost function may alternatively or additionally be arranged to reflect the difference between the training multi-channel audio signal and the multi-channel audio signal generated by upmixing the downmix audio signal and the generated auxiliary audio signal. In such an example, the downmixer 601 may also generate upmix parameter data that can be used in the upmix. Therefore, in some embodiments, instead of only training the artificial neural network to generate the auxiliary audio signal that matches the residual signal, the sequence may include reconstructing the multi-channel audio signal based on the generated auxiliary audio signal. The generation of the output multi-channel audio signal may specifically be performed using the same operation performed in the encoder. This output multi-channel audio signal may then be compared with the input multi-channel audio signal.

[0160] In some embodiments, the cost function may also take into account other parameters. For example, in some embodiments, the cost function may also include consideration of the degree of correlation between the generated auxiliary audio signal and the downmix signal. Specifically, the cost function may indicate that the higher the decorrelation of the downmix audio signal and the auxiliary audio signal, the lower the cost. An increase in decorrelation may indicate that there is less common information in the two signals.

[0161] In an embodiment where sub-band operations are performed, the described method may be performed for each sub-band. Specifically, a residual audio signal may be generated for each sub-band and compared with the generated auxiliary audio signal sub-band samples for that sub-band. Thus, the cost function may be evaluated for each sub-band to determine a cost value, and the coefficients of the feature set artificial neural network for that sub-band and the auxiliary sub-band artificial neural network may be trained.

[0162] In many embodiments, the feature set artificial neural network and / or indeed the first artificial neural network 107 may generate a set of feature values ​​that matches the time and frequency resolution of the auxiliary sub-band artificial neural network, and in particular the second artificial neural network 109 if only one artificial neural network is used. However, in other embodiments, the time and / or frequency resolution of the generated set of feature values ​​may be different, and in general the resolution for the set of feature values ​​may be lower. In such cases, a function to change the resolution may be included. In particular, an interpolator may be incorporated to generate an interpolated set of parameter values ​​from the generated parameter values.

[0163] In some embodiments, one or more of the sets of feature values ​​may include one or more values ​​generated in other ways in addition to the feature values ​​generated by the feature set artificial neural network.

[0164] In fact, in some embodiments, feature values ​​reflecting user input may be included. Specifically, Figure 1 , Figure 4 and Figure 5 The apparatus in may include a user input and a user input processor, which may be arranged to generate a feature value in response. This method may allow a user to control or adjust the generation operation of the multi-channel audio signal. For example, it may allow the selection of a specific audio mode, which may affect the reconstructed multi-channel audio signal via the generation of an auxiliary audio signal. The user input may be manually related to the perceived audio quality and made by an expert. These additional terms may reduce the quantitative reconstruction of the audio, for example expressed as RMSE, but improve the human perception of the audio reconstruction.

[0165] For example, an auxiliary sub-band artificial neural network can be trained based on the feature set. For example, if the feature value may have multiple discrete possible values, each possible value corresponds to a user setting. Each of these settings may correspond to a desired change in the multi-channel audio signal. For example, one user input may increase the low and high frequencies, another user input may be filtered to provide only the mid-frequency, and a third setting may have a large amount of reverberation or echo. During training, the reference multi-channel audio signal used for comparison in the cost function can be processed according to specific preferences for different settings. The network can then be trained for all possible settings of the user pattern feature value, but the reference multi-channel audio signal is selected as a reference for processing for that specific user pattern feature value.

[0166] As another example, feature values ​​may be generated in response to different modalities. For example, facial detection may be used to identify a user, and the feature values ​​may be set to reflect the current user. This may allow automatic adaptation to individual users.

[0167] In many embodiments, one or more of the feature values ​​may be generated by analysis of the downmix audio signal.

[0168] It will be appreciated that many different algorithms and procedures are known for analysing audio signals to extract or determine properties of the audio signals, and that any suitable (analysis) methods, algorithms and properties may be used.

[0169] The audio device(s) may in particular be implemented in one or more suitable processor(s). In particular, the artificial neural network may be implemented in one or more such suitably programmed processors. Different functional blocks, and in particular the artificial neural network, may be implemented in separate processors and / or may be implemented, for example, in the same processor. Examples of suitable processors are provided below.

[0170] Figure 7 is a block diagram illustrating an example processor 700 according to an embodiment of the present disclosure. The processor 700 may be used to implement one or more processors that implement the apparatus or elements thereof as described above (particularly including one or more artificial neural networks). The processor 700 may be any suitable processor type, including but not limited to a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable array (FPGA), where the FPGA has been programmed to form a processor, a graphics processing unit (GPU), an application specific circuit (ASIC), where the ASIC is designed to form a processor, or a combination thereof.

[0171] Processor 700 may include one or more cores 702. Core 702 may include one or more arithmetic logic units (ALUs) 704. In some embodiments, core 702 may include a floating point logic unit (FPLU) 706 and / or a digital signal processing unit (DPU) 708 in addition to or in lieu of ALU 704.

[0172] The processor 700 may include one or more registers 312 communicatively coupled to the core 702. The registers 712 may be implemented using dedicated logic gates (e.g., flip-flops) and / or any memory technology. In some embodiments, the registers 712 may be implemented using static memory. The registers may provide data, instructions, and addresses to the core 702.

[0173] In some embodiments, the processor 700 may include one or more levels of cache memory 710 communicatively coupled to the core 702. The cache memory 710 may provide computer readable instructions to the core 702 for execution. The cache memory 710 may provide data for processing by the core 702. In some embodiments, the computer readable instructions may have been provided to the cache memory 710 by a local memory (e.g., a local memory attached to the external bus 716). The cache memory 710 may be implemented with any suitable cache memory type, such as a metal oxide semiconductor (MOS) memory, such as a static random access memory (SRAM), a dynamic random access memory (DRAM), and / or any other suitable memory technology.

[0174] The processor 700 may include a controller 714 that may control inputs to the processor 700 from other processors and / or components included in the system and / or outputs from the processor 700 to other processors and / or components included in the system. The controller 714 may control data paths in the ALU 704, FPLU 706, and / or DSPU 708. The controller 714 may be implemented as one or more state machines, data paths, and / or dedicated control logic. The gates of the controller 714 may be implemented as independent gates, FPGAs, ASICs, or any other suitable technology.

[0175] Registers 712 and cache 710 may communicate with controller 714 and core 702 via internal connections 720A, 720B, 720C, and 720D. The internal connections may be implemented as a bus, multiplexers, crossbar switches, and / or any other suitable connection technology.

[0176] Input and output of the processor 700 may be provided via a bus 716, which may include one or more conductors. The bus 716 may be communicatively coupled to one or more components of the processor 700, such as the controller 714, cache 710, and / or registers 712. The bus 716 may be coupled to one or more components of the system.

[0177] The bus 716 may be coupled to one or more external memories. The external memory may include a read-only memory (ROM) 732. The ROM 732 may be a mask ROM, an electrically programmable read-only memory (EPROM), or any other suitable technology. The external memory may include a random access memory (RAM) 733. The RAM 733 may be a static RAM, a battery-backed static RAM, a dynamic RAM (DRAM), or any other suitable technology. The external memory may include an electrically erasable programmable read-only memory (EEPROM) 735. The external memory may include a flash memory 734. The external memory may include a magnetic storage device, such as a disk 736. In some embodiments, the external memory may be included in the system.

[0178] The present invention can be implemented in any suitable form, including hardware, software, firmware or any combination of these. The present invention can be optionally implemented at least in part as computer software running on one or more data processors and / or digital signal processors. The elements and components of an embodiment of the present invention can be implemented physically, functionally and logically in any suitable manner. In fact, the function can be implemented in a single unit, in multiple units or as a part of other functional units. Like this, the present invention can be implemented in a single unit, or can be physically and functionally distributed between different units, circuits and processors.

[0179] Although the present invention has been described in conjunction with some embodiments, it is not intended to limit the present invention to the specific forms set forth herein. On the contrary, the scope of the present invention is limited only by the appended claims. In addition, although it may appear that features have been described in conjunction with specific embodiments, those skilled in the art will recognize that the various features of the described embodiments may be combined according to the present invention. In the claims, the term "comprising" does not exclude the presence of other elements or steps.

[0180] In addition, although listed separately, multiple devices, elements, circuits or method steps can be implemented by, for example, a single circuit, unit or processor. In addition, although individual features may be included in different claims, these features may be advantageously combined, and inclusion in different claims does not mean that the combination of features is not feasible and / or disadvantageous. The inclusion of features in a class of claims does not mean a limitation on the class, but rather indicates that the feature is equally applicable to other claim classes when appropriate. In addition, the order of features in the claims does not mean any specific order in which the features must work, and in particular, the order of the individual steps in the method claims does not mean that the steps must be performed in that order. Instead, the steps can be performed in any suitable order. In addition, singular references do not exclude multiples. Therefore, references to "one", "one", "first", "second", etc. do not exclude multiples. The figure marks in the claims are provided only to make the examples clear and should not be interpreted as limiting the scope of the claims in any way.

Claims

1. A device for generating a multi-channel audio signal, the device comprising: a receiver (101) arranged to receive a downmix audio signal for the multi-channel audio signal and upmix parameter data for upmixing the downmix audio signal; a first artificial neural network (107) arranged to generate a set of feature values ​​for the downmix audio signal, the first artificial neural network having an input node for receiving a first sample of the downmix audio signal and an output node for providing the set of feature values; a second artificial neural network (109) having an input node for receiving a second sample of the downmix audio signal and an output node arranged to provide a sample of the auxiliary audio signal for the downmix audio signal, the second artificial neural network (109) further comprising a node receiving a feature value from the set of feature values; as well as A generator (105) is arranged to generate the multi-channel audio signal from the downmix signal and the auxiliary audio signal in dependence on the upmix parameter data.

2. The apparatus according to claim 1, comprising a first filter bank (401) for generating a frequency sub-band representation of the downmix audio signal; and wherein At least some of the second samples of the downmix audio signal are sub-band samples of the frequency sub-band representation.

3. The device according to claim 2, wherein: The second artificial neural network (109) is an artificial neural network of a first plurality of sub-band artificial neural networks, each sub-band artificial neural network of the first plurality of sub-band artificial neural networks being arranged to generate sub-band samples for a subset of sub-bands of a frequency sub-band representation of the auxiliary audio signal.

4. The device according to claim 3, wherein: The plurality of sub-band artificial neural networks comprises an artificial neural network for each sub-band of the frequency sub-band representation of the auxiliary audio signal.

5. The device according to any one of claims 2 to 4, wherein: The generator (105) is arranged to generate a frequency subband representation of the multichannel audio signal by applying a subband matrix operation to the frequency subband representation of the auxiliary audio signal and the frequency subband representation of the downmix audio signal, and is arranged to transform the frequency subband representation of the multichannel audio signal into a time domain representation of the multichannel audio signal.

6. The device according to any one of the preceding claims 2 to 5, wherein: The set of feature values ​​generated by a sub-band artificial neural network of the first plurality of sub-band artificial neural networks is common to a plurality of sub-bands of the frequency sub-band representation of the downmix audio signal.

7. The device according to any one of the preceding claims 2 to 6, wherein: The number of input nodes of the artificial neural network in the first plurality of sub-band artificial neural networks decreases monotonically with increasing frequency.

8. The apparatus according to any of the preceding claims, comprising a second filter bank (401) for generating a frequency sub-band representation of the downmix audio signal; and wherein At least some of the first samples of the downmix audio signal are sub-band samples of the frequency sub-band representation.

9. The device according to claim 8 when dependent on any one of claims 3 to 7, wherein: The first artificial neural network (107) is an artificial neural network of a second plurality of sub-band artificial neural networks, each sub-band artificial neural network of the second plurality of sub-band artificial neural networks being arranged to generate sub-band samples for a subset of the artificial neural networks of the first plurality of artificial neural networks.

10. Apparatus according to claim 9 when dependent on any one of claims 3 to 7, wherein: The sub-band samples in the second sub-band samples for at least one artificial neural network in the second plurality of artificial neural networks include a plurality of sub-band samples for a plurality of processing time intervals for an artificial neural network in the first plurality of artificial neural networks.

11. The device according to claim 9 or 10, wherein: The subband samples in the second subband samples for at least one artificial neural network of the second plurality of artificial neural networks include at least one subband sample for a subband of the subband representation of the downmix audio signal, for which the at least one artificial neural network does not generate subband samples for the subband representation of the auxiliary audio signal.

12. An apparatus according to any preceding claim, wherein: The first artificial neural network and the second artificial neural network are trained by a joint training process based on training data and using a cost function, wherein the training data includes: a set of samples of a downmix audio signal generated by downmixing a training multi-channel audio signal, and a target audio signal determined based on a residual signal generated for the downmix audio signal, and the cost function indicates a difference between an auxiliary audio signal generated for the training multi-channel audio signal and an auxiliary audio signal generated for the target audio signal.

13. An apparatus according to any preceding claim, further comprising generating eigenvalues ​​for the set of eigenvalues ​​based on an analysis of the downmix audio signal.

14. A method for generating a multi-channel audio signal, the method comprising: receiving a downmix audio signal for the multi-channel audio signal and upmix parameter data for upmixing the downmix audio signal; A first artificial neural network (107) generates a set of feature values ​​for the downmix audio signal, the first artificial neural network having an input node for receiving a first sample of the downmix audio signal and an output node for providing the set of feature values; a second artificial neural network (109) having an input node for receiving a second sample of the downmix audio signal and an output node for providing a sample of the auxiliary audio signal for the downmix audio signal, the second artificial neural network (109) further comprising a node receiving a feature value from the set of feature values; as well as The multi-channel audio signal is generated from the downmix signal and the auxiliary audio signal in dependence on the upmix parameter data.

15. A computer program product comprising computer program code means adapted to perform all the steps of claim 14 when said program is run on a computer.

Images