Stereo decoding method of neural network model and decoder

Abstract

The invention relates to a method and a decoder for reconstructing a stereo audio signal. The method comprises: receiving (S1) a bitstream comprising an encoded first monaural audio signal and a set of reconstruction parameters; and decoding the encoded first monaural audio signal to provide a first monaural audio signal. The method further comprises reconstructing (S3a) the second monophonic audio signal using a neural network system (24, 24a, 24b, 24c, 24d, 24e) trained to predict the second monophonic audio signal given samples of the first monophonic audio signal and the reconstruction parameters, or reconstructing (S3a, S3b) the second monophonic audio signal using a neural network system (24c, 24d, 24e), the neural network system (24, 24a, 24b, 24c, 24d, 24e) trained to predict the second monophonic audio signal given samples of the first monophonic audio signal and the reconstruction parameters. S3b) the second monaural audio signal and the third monaural audio signal, the neural network system being trained to predict the second monaural audio signal and the third monaural audio signal given the samples of the first monaural audio signal and the reconstruction parameters.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims the benefit of priority of U.S. Provisional Application Serial No. 63 / 433,737, filed on Dec. 19, 2022, and European Patent Application No. 23157900.4, filed on Feb. 22, 2023, each of which is hereby incorporated by reference in its entirety. Technical Field

[0003] This application relates to a method and a decoder for stereo decoding, and particularly to stereo reconstruction using a neural network model. Background Art

[0004] Stereo audio is used to present various types of audio content (e.g., music), and is suitable for rendering via headphones, stereo speaker pairs, or even surround sound speaker devices with more than two speakers using various upmixing techniques. Stereo audio consists of two audio signals, e.g., a left audio signal and a right audio signal, which together form a stereo pair. The left audio signal and the right audio signal can be recorded using two spatially offset microphones and / or using two directional microphones pointing in different directions (e.g., at a 90-degree angle). Although only two audio signals are involved, stereo audio can be used to create an immersive three-dimensional spatial effect, giving the listener a sense of direction in the rendered audio scene. For example, a user listening to stereo audio via headphones can perceive the source of the audio content to be located somewhere between the listener's two ears (the source moves with the panning of the audio signal), or the source to be outside the user (the source moves with the left and right audio signals provided with relative delays or with processing using a head-related transfer function (HRTF)).

[0005] Other audio signal pairs other than the left and right audio signal pair can also be used to represent stereo audio. For example, stereo audio can be represented by a mid-side stereo pair formed by a mid audio signal and a side audio signal. There are many ways to capture or create a mid-side audio pair. For example, a left-right stereo pair can be converted to a mid-side stereo pair, or a mid-side stereo pair can be recorded using an omnidirectional or forward-facing microphone (recording the mid audio signal) and a side-facing microphone (recording the side audio signal). One benefit of a mid-side stereo pair is that the mid audio signal typically captures the most critical audio content, making the mid-side stereo pair backward compatible with mono playback systems that ignore the side audio signal and only render the mid audio signal.

[0006] For the same reason, mid-side stereo pairs are typically used when performing stereo encoding and stereo decoding. A stereo audio signal including two audio signals carries more information than a mono audio signal, which means that, for example, transmitting a stereo audio signal requires a higher bit rate and storing a stereo audio signal requires a larger amount of data. To this end, several encoders have been proposed that obtain a left-right stereo pair, convert it to a mid-side stereo pair, and encode the mid audio signal as a downmixed audio signal that is sent to a decoder together with some side parameters indicating the correlation between the left and right audio signals. Thus, only the downmixed mid audio signal (mid audio signal) is encoded and sent together with the side parameters (including a relatively small amount of data), which means that the encoded stereo representation is highly compressed. The decoder decodes the downmixed mid audio signal and, guided by the side parameters, converts it to a left-right stereo pair.

[0007] Although this type of encoding and decoding is not lossless, it has proven to be efficient in providing a high level of compression while retaining a high perceptual quality of the reconstructed stereo pair. In some techniques, the downmixed mid audio signal is passed through an all-pass filter to generate a synthetic side signal from the downmixed mid audio signal, and the selection of the filter parameters of the all-pass filter introduces a fixed time delay. It has been proven that an all-pass filter with a fixed delay is a suitable method for generating a synthetic but convincing side audio signal from the mid audio signal, where the temporal and spectral energy distribution of the side audio signal is approximately the same as that of the downmixed mid audio signal. Then, the downmixed mid audio signal and the synthetic side audio signal are used together with the side parameters to convert the mid-side stereo pair to a left-right stereo pair. SUMMARY OF THE INVENTION

[0008] One problem with the above prior art techniques is that the decoding process fails to reproduce a convincing stereo pair when the original left audio signal and the original right audio signal are strongly decorrelated. Examples of strongly decorrelated audio include audio signals representing rain, applause, or even certain types of music. To this end, an improved method for decoding encoded stereo audio is needed that overcomes at least some of the disadvantages mentioned above.

[0009] According to a first aspect of the present invention, there is provided a method for reconstructing a stereo audio signal. The method comprises the steps of: receiving a bitstream comprising an encoded first mono audio signal and a set of reconstruction parameters; and decoding the encoded first mono audio signal to provide the first mono audio signal. The method further comprises: reconstructing a second mono audio signal using a neural network system, the neural network system being trained to predict samples of the second mono audio signal given samples of the first mono audio signal and the reconstruction parameters, wherein the first mono audio signal and the reconstructed second mono audio signal form a stereo audio signal pair.

[0010] A stereo audio signal pair refers to two audio signals that together form a stereo format. For example, the two audio signals of a stereo audio signal pair can be recorded using two microphones in a stereo re-recording configuration. A stereo audio signal pair may also be generated during a mixing process. The most common format of a stereo audio signal pair is a left-right stereo audio signal. However, there are many alternative formats of stereo audio signal pairs, such as a center stereo audio signal and a side stereo audio signal.

[0011] The bitstream is an encoded representation of the original stereo audio signal. By including a single mono audio signal and reconstruction parameters in the bitstream, an efficient and highly compressed representation is achieved, which facilitates the transmission of the stereo audio signal or storage in a data storage medium. Although the encoding process for achieving such a representation is generally not lossless, it has been proven that it can accurately reconstruct the original stereo audio signal pair with a high perceptual quality.

[0012] The present invention is at least partially based on the understanding that a trained neural network model will be able to reconstruct the second mono audio signal with higher quality compared to the second mono audio signal that has been analyzed and calculated in a traditional decoder. In particular, when the correlation between the audio signals of the original stereo audio signal pair is low, the encoded representation cannot carry enough information to accurately reconstruct the second mono audio signal, which results in poor performance of the traditional decoder. On the other hand, by using a trained neural network model, a second mono audio signal with higher perceptual quality can be reconstructed even when the correlation between the original stereo audio signals is low or non-existent. Therefore, an efficient and highly compressed bitstream can still be used even when the correlation between the original stereo audio signals is low.

[0013] According to a second aspect of the present invention, there is provided a method for reconstructing a stereophonic audio signal, the method comprising the steps of: receiving a bitstream comprising an encoded first mono audio signal and a set of reconstruction parameters; and decoding the encoded first mono audio signal to provide the first mono audio signal. The method further comprises: reconstructing a second mono audio signal and a third mono audio signal using a neural network system, the neural network system being trained to predict samples of the second mono audio signal and samples of the third mono audio signal given samples of the first mono audio signal and the reconstruction parameters, wherein the reconstructed second mono audio signal and the reconstructed third mono audio signal form a stereophonic audio signal pair.

[0014] That is, as an alternative to a neural network model configured as a single output network and trained to reconstruct a second mono audio signal that forms a stereophonic audio signal pair with the first mono audio signal, the neural network model can be configured as a dual output network and trained to directly reconstruct the second mono audio signal and the third mono audio signal, wherein the second mono audio signal and the third mono audio signal form a stereophonic audio signal pair. The second aspect of the present invention has the same or equivalent advantages as the first aspect of the present invention.

[0015] Additionally, the method of the second aspect of the present invention enables the neural network model to introduce additional enhancement when reconstructing a stereophonic audio signal pair. For example, the second mono audio signal can still form a stereophonic audio signal pair with the first mono audio signal, but the third mono audio signal is an enhanced version of the first mono audio signal, which is predicted by the neural network model and provides enhanced perceptual quality.

[0016] As another example, the first mono audio signal can have a stereophonic audio signal format (e.g., mid-side format) different from the desired output of the decoder (e.g., left format and right format). Thus, with the dual output network, the second mono audio signal and the third mono audio signal can have a desired stereophonic audio signal format (e.g., left format and right format) different from the stereophonic audio format of the first mono audio signal.

[0017] In some embodiments of the first aspect and / or the second aspect of the present invention, the neural network system is trained to operate on flattened audio signal samples, and the method further comprises: performing envelope flattening on the first mono audio signal to produce a flattened first mono audio signal and providing the flattened first mono audio signal to the neural network system. The method further comprises at least inverse flattening the reconstructed second mono audio signal.

[0018] Therefore, the neural network model is trained to operate on the flattened samples of the first mono audio signal. By operating on the flattened samples, the performance of the neural network model can be enhanced, while also allowing the use of a neural network model with lower complexity that is easier to train. In most audio content, the spectral energy content of the low frequencies is higher than that of the high frequencies, i.e., the audio content has a high dynamic range. If the samples of the audio signal are not flattened, the neural network model will inherently prioritize the accurate reconstruction of the low frequencies over the accurate reconstruction of the high frequencies, which may result in significant distortion or a decrease in the quality of the reconstructed audio signal for certain types of audio content. By flattening the samples, the spectral energy content will be more evenly distributed across all frequencies, which means that the neural network model will equally prioritize the accurate reconstruction of all frequencies, thereby improving the quality of the reconstructed audio signal.

[0019] According to a third aspect of the present invention, there is provided a computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method according to the first aspect or the second aspect of the present invention.

[0020] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium storing a computer program according to the third aspect of the present invention.

[0021] According to a fifth aspect of the present invention, there is provided a decoder comprising a processor and a memory coupled to the processor, wherein the processor is adapted to perform the steps of the method according to the first aspect or the second aspect of the present invention.

[0022] According to a sixth aspect of the present invention, there is provided a method for training a neural network for stereo reconstruction. The method includes: obtaining training data including a pair of stereo audio signals; and encoding the pair of stereo audio signals into an encoded stereo audio signal including a first mono audio signal and reconstruction parameters. The method further includes: reconstructing a second mono audio signal using a neural network system trained to predict samples of the second mono audio signal given samples of the first mono audio signal and the reconstruction parameters; and determining a difference metric between the reconstructed second mono audio signal and a ground truth second mono audio signal associated with the pair of stereo audio signals of the training data. Finally, the method includes modifying the internal weights of the neural network model based on the determined difference.

[0023] The method for training a neural network according to the sixth aspect is applicable to training the neural network model according to the first aspect of the present invention.

[0024] In some embodiments, the neural network model is further configured to predict samples of a third mono audio signal given samples of a first mono audio signal and the reconstruction parameters, and the method further includes reconstructing the third mono audio signal using the neural network model and determining a difference metric between the reconstructed third mono audio signal and a ground truth third mono audio signal associated with a stereo audio signal pair of the training data.

[0025] Embodiments of this training method are applicable to training the neural network model used in the second aspect of the present invention.

[0026] The third to sixth aspects of the present invention have the same or equivalent advantages as the first and second aspects of the present invention. Any function described with respect to the method can have a corresponding feature in the system, and vice versa. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present invention will be described in more detail with reference to the accompanying drawings, which show currently preferred embodiments of the present invention.

[0028] FIG. 1a depicts a stereo encoder sending an encoded stereo bitstream to a stereo decoder according to some embodiments.

[0029] FIG. 1b depicts a detailed view of a stereo encoder according to some embodiments.

[0030] FIG. 2a depicts a detailed view of a stereo decoder according to some embodiments.

[0031] FIG. 2b depicts a detailed view of another stereo decoder according to some embodiments.

[0032] FIG. 2c depicts a detailed view of a stereo decoder having a dual-output neural network model according to some embodiments.

[0033] FIG. 2d depicts a detailed view of another stereo decoder having a dual-output neural network model and performing stereo format conversion according to some embodiments.

[0034] FIG. 3a depicts a single-output neural network model according to some embodiments.

[0035] FIG. 3b depicts a single-output neural network model operating on flattened samples according to some embodiments.

[0036] FIG. 3c depicts a dual-output neural network model operating on flattened samples according to some embodiments.

[0037] FIG. 3d depicts a dual-output neural network model operating on flattened samples with stereo format conversion according to some embodiments.

[0038] FIG. 3e depicts a dual-output neural network model for converting a stereo format to an alternative format according to some embodiments.

[0039] FIG. 4a is a flowchart depicting a method for decoding a stereo audio signal using a single-output neural network model according to some embodiments.

[0040] FIG. 4b is a flowchart depicting a method for decoding a stereo audio signal using a dual-output neural network model according to some embodiments.

[0041] FIG. 5a depicts a training setup for training a neural network model according to some embodiments.

[0042] FIG. 5b is a flowchart depicting a method for training a neural network model according to some embodiments.

[0043] Figure 6 The figure illustrates a neural network model according to some embodiments, in which one of the neural network model and the LTI filter is selectively used based on the content of the first mono audio signal. DETAILED DESCRIPTION

[0044] The systems and methods disclosed in this application can be implemented as software, firmware, hardware, or a combination thereof. In a hardware implementation, the division of tasks does not necessarily correspond to the division of physical units; rather, one physical component can have multiple functions, and one task can be executed collaboratively by several physical components.

[0045] Computer hardware can be, for example, a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a smartphone, a web device, a network router, a switch, or a bridge, or any machine capable of executing instructions (sequentially or otherwise) specifying actions to be taken by that computer hardware. Further, the present disclosure will relate to any collection of computer hardware that individually or jointly executes instructions to perform any one or more of the concepts discussed herein.

[0046] Some or all of the components may be implemented by one or more processors that receive computer-readable (also referred to as machine-readable) code containing a set of instructions that, when executed by the one or more processors, perform at least one of the methods described herein. Any processor that includes a set of instructions capable of (sequentially or otherwise) performing the specified actions to be taken. Thus, an example is a typical processing system (i.e., computer hardware) that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem that includes a hard drive, an SSD, RAM, and / or ROM. A bus subsystem may be included for communication between components. Software may reside in the memory subsystem and / or within the processor during its execution by the computer system.

[0047] The one or more processors may operate as a stand-alone device or may be connected to (multiple) other processors, such as network-connected to (multiple) other processors. Such a network may be built on various different network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0048] Software may be distributed on computer-readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, various forms of physical (non-transitory) storage media such as EEPROM, flash memory, or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical disk storage devices, magnetic tape cartridges, tapes, magnetic disk storage devices, or any other media that can be used to store the desired information and can be accessed by a computer. Further, as is well known to those skilled in the art, (transitory) communication media typically embodies computer-readable instructions, data structures, program modules, or other data in the form of a modulated data signal such as a carrier wave or other transmission mechanism, and includes any information delivery media.

[0049] In FIG. 1a, an encoder 10 and a decoder 20 for encoding and decoding a stereo audio signal pair are presented. The original left audio signal L and the original right audio signal R that form the stereo audio signal pair are provided to the encoder 10, which encodes the original stereo signal pair L, R into an encoded signal representation that is included in the bitstream B. Transforming the original stereo signal pair L, R into the encoded representation may be a lossy process, in which some information present in the original stereo audio signal pair is omitted. For example, the encoder 10 omits one of the original left audio signal L and the original right audio signal R and includes the other of the original left audio signal L and the original right audio signal R in the bitstream B. The encoder 10 further extracts a reconstruction parameter indicating the relationship (e.g., covariance) between the original left audio signal L and the original right audio signal R, where the reconstruction parameter is also included in the bitstream B. The bitstream B is then provided to the decoder 20, which reconstructs the left audio signal L * and / or the reconstructed right audio signal R * from the content of the bitstream B. In some embodiments, the decoder 20 reconstructs the reconstructed left audio signal L * and / or the reconstructed right audio signal R * both. Alternatively, the decoder 20 reconstructs an audio signal complementary to the audio signal included in the bitstream B, i.e., reconstructs only one of the reconstructed left audio signal L * and the reconstructed right audio signal R * .

[0050] In some embodiments, the encoder 10 transforms the original left audio signal L and the original right audio signal R into a mid-side stereo format and includes only the mid audio signal together with a reconstruction parameter indicating the attributes of the side audio signal in the bitstream B. Since the mid audio signal is expected to capture the most critical information in the stereo signal pair (e.g., most stereo audio signals are centered panned, meaning that most of the spectral energy will be included in the mid audio signal), including the mid audio signal in the bitstream B instead of one of the left audio signal L and the right audio signal R enables a more accurate reconstruction in the decoder 20. The decoder 20 then uses the content in the bitstream B to reconstruct the reconstructed side signal or the reconstructed left audio signal L * and the reconstructed right audio signal R * .

[0051] Referring further to FIG. 1b, the operation of the encoder 10 will now be described in more detail. The original left and right stereo signal pair L, R are received and provided to the stereo downmix unit 12. The stereo downmix unit 12 performs two tasks: extracting a first mono audio signal α (e.g., in the form of a mid-audio signal M), also referred to as a downmixed audio signal; and extracting reconstruction parameters P that indicate attributes of the relationship between the original left audio signal L and the original right audio signal R. Extracting the mid-audio signal M from the left audio signal L and the right audio signal R is achieved, for example, by the following equation:

[0052] M = g l L + g r R (1)

[0053] where g l and g r are channel weights, and setting g l and g r equal to 1 / 2 produces a conventional mono audio signal. Similarly, the corresponding side audio signal S can be created as

[0054] S = g l L - g r R. (2)

[0055] It is also conceivable that the encoder 10 extracts a target mid-audio signal, where the weights g l and g r vary with time and / or frequency. For example, the mid-audio signal and the side audio signal are determined as:

[0056] M = cos(Θ)L + sin(Θ)R (3)

[0057] S = sin(Θ)L - cos(Θ)R (4) where g l = cos(Θ) and g r = sin(Θ), and where Θ varies with time and optionally with frequency. The parameter Θ is referred to as the target pan parameter Θ, and ranges from 0 to since it determines the panning of the target audio source in the stereo pair L, R, and the resulting dynamic mid-audio signal and dynamic side audio signal are referred to as the target mid-audio signal and the target side audio signal.

[0058] In this objective, the audio signal in the target and the target-side audio signal are related to the time-varying sound image panning determined by the target sound image panning parameter Θ with respect to the left and right stereo pair. To this end, the target sound image panning parameter Θ can be sent together with the reconstruction parameter P and used by the neural network model of the decoder and / or the mixer when reconstructing the stereo audio signal in left and right formats. For example, the target sound image panning parameter Θ varies with time and frequency to extract the audio signal in the target that captures the dominant audio source in each frequency band. To this end, the target sound image panning parameter Θ can be set to the estimated sound image panning in each frequency band. For example, the target sound image panning parameter Θ is calculated as arctan(|R| / |L|), where |R| and |L| are the spectral energies of the left audio signal L and the right audio signal R for a specific time period and frequency band. For example, if the left signal contains the maximum spectral energy for a specific frequency band and time period, then Θ = arctan(|R| / |L|) approaches 0, which means that the audio signal in the target will be dominated by the left channel.

[0059] In addition, it is conceivable that a target phase difference parameter Φ can be obtained for each time period and frequency band of the left and right stereo pair. For example, for each time period and frequency band, the target phase difference parameter Φ is determined as Φ = Arg(R / L). Then, the target phase difference parameter Φ can be used together with the target sound image panning parameter Θ to extract the audio signal in the target and the target-side audio signal based on both the target sound image panning parameter Θ and the target phase difference parameter Φ, as follows:

[0060]

[0061] Thus, the target sound image panning parameter Θ and the target phase difference parameter Φ are sent together with the reconstruction parameter P and used by the neural network model of the decoder and / or the mixer when reconstructing the stereo audio signal in left and right formats.

[0062] Therefore, while traditional audio signal capture in the center has no phase difference for panning the audio source, the audio signal in the target obtained through Equation 3 or Equation 5 above will dynamically target the source whose frequency, sound image panning, and phase vary with time in the left and right stereo audio signals, so that the most prominent audio source is always present in the audio signal in the target. In other words, with the audio signal in the target, even if the sound image panning, phase, or frequency of the dominant audio source changes with time, the risk that the stereo mix does not include the dominant source can be reduced.

[0063] The extraction and utilization of the target pan parameter Θ and / or the target phase difference parameter Φ, and the reconstruction of the left and right stereo audio signals from the target mid audio signal and the target side audio signal are described in more detail in U.S. Provisional Application No. 63 / 318,226, "TARGET MID-SIDE SIGNALS FOR AUDIO APPLICATIONS," filed on March 9, 2022, which is hereby incorporated by reference in its entirety.

[0064] Extracting the reconstruction parameter P can involve extracting at least one of the inter-channel intensity difference (IID), the inter-channel cross-correlation (ICC), the inter-channel phase difference (IPD), and the inter-channel time difference (ITD) of the original left and right audio signal channels.

[0065] The inter-channel intensity difference or IID indicates the intensity difference between the two signals in the original stereo signal pair L, R.

[0066] The inter-channel cross-correlation or ICC indicates the cross-correlation or coherence between the two signals in the original stereo signal pair L, R. In some embodiments, the coherence is determined as the maximum value of the cross-correlation as a function of time or phase.

[0067] The inter-channel phase difference or IPD indicates the phase difference between the two signals in the original stereo signal pair L, R. An alternative to IPD is the inter-channel time difference or ITD, which indicates the time difference between the two signals of the original stereo audio signals L, R.

[0068] It is also contemplated that the reconstruction parameter indicates the target pan parameter Θ and / or the target phase difference parameter Φ for each time period and frequency band. This allows the neural network model of the decoder and / or the mixer to reconstruct the original left and right audio signals from the target mid audio signal extracted using the target pan parameter Θ and / or the target phase difference parameter Φ.

[0069] In some embodiments, a first mono audio signal α is provided to a mono signal encoder 13, which is configured to encode the first mono audio signal α into an encoded first mono audio signal E(α). The encoding performed by the mono signal encoder can be lossless or lossy. Lossy encoding enables the first mono audio α to be compressed. For example, the mono signal encoder 13 can perform downsampling or quantization on the first mono audio signal α. Although the bitstream encoder 11 is depicted as separate from the encoder 10, the bitstream encoder 11 can also be part of the encoder 10, which then receives a pair of stereo audio signals L, R as input and outputs an encoded bitstream B.

[0070] In some embodiments, the reconstruction parameter P is also compressed using quantization, e.g., performed by quantizer 14.

[0071] (Optionally encoded) The first mono audio signal and (optionally encoded) the reconstruction parameter P are provided to the bitstream encoder 11, which encodes the information into a bitstream B. Then the bitstream B is stored or transmitted (e.g., over a network) to the decoder 20.

[0072] As shown in FIG. 2a, the bitstream B is received by the bitstream decoder 21, which decodes the bitstream B to obtain the first mono audio signal α and the reconstruction parameter P contained in the bitstream B. The bitstream decoder 21 can be provided separately from or integrated with the stereo decoder 20. The bitstream decoder 21 decodes the bitstream encoding and any encoding that encapsulates the first mono audio signal α and the reconstruction parameter P, and provides the first mono audio signal α and the reconstruction parameter P to the neural network model 24a of the stereo decoder 20.

[0073] In some embodiments, samples of the first mono audio signal α and the reconstruction parameter P are provided as input parameters to the neural network model 24a, which is trained to predict samples of the reconstructed second mono audio signal β*. The first mono audio signal α and the reconstructed second mono audio signal β* form a stereo audio signal pair. For example, the first mono audio signal α is the mid audio signal, and the reconstructed second mono audio signal β* is the side audio signal, where the two audio signals form a mid-side stereo audio signal pair.

[0074] Optionally, if the first mono audio signal α and the reconstructed second mono audio signal β* are not yet in the left-right stereo audio signal format, the first mono audio signal α and the reconstructed second mono audio signal β* are provided to the mixing unit 26, which mixes the first mono audio signal α and the reconstructed second mono audio signal β to form a reconstructed left-right stereo audio signal pair L*, R*. For example, the mixing unit 26 is provided with a target pan parameter Θ x and uses this parameter Θ x to reconstruct the left audio signal L* and the right audio signal R*.

[0075] The neural network model 24a can include any type of neural network. For example, the neural network is a recurrent neural network (RNN) or a convolutional neural network (CNN). In some embodiments, the neural network can include multiple neural network layers.

[0076] The neural network model 24a may include a generative model. A generative model is a neural network that implements a probability distribution (e.g., a conditional probability distribution) that models the probability distribution of the dataset on which the neural network is trained. The reconstruction of the second (and optional third) monaural audio signal β* is achieved by randomly sampling according to the probability distribution implemented by the trained neural network.

[0077] The architecture of the generative model may be similar to, for example, the generative model described in detail in U.S. Provisional Application No. 63 / 331,056, "HIGH FREQUENCY RECONSTRUCTION USING NEURAL NETWORK SYSTEM," filed on April 14, 2022, which is hereby incorporated by reference in its entirety. The generative model uses a neural network system to reconstruct the filter bank domain high-frequency band signal. The neural network system is trained to predict samples of the high-frequency band audio signal in the filter bank domain given samples of the filter bank domain low-frequency band signal and high-frequency reconstruction (HFR) parameters, where the HFR parameters describe the attributes of the higher frequency band. In one example, the neural network system includes an upper neural network and a lower neural network. In the upper neural network, previously generated filter bank samples are received together with decoded low-frequency band samples and high-frequency reconstruction parameters. The lower neural network is divided into a plurality of sequentially executed sub-layers, each sub-layer being configured to generate a set of channels of the reconstructed high-frequency band. In some embodiments, the generative model also reconstructs an enhanced low-frequency band audio signal.

[0078] When applying this generative model for high-frequency reconstruction to audio signal reconstruction, the difference lies in the size of the output, which depends on whether the neural network model generates a single monaural audio signal as output or two monaural audio signals as output. It is conceivable that by providing the first monaural audio signal (instead of the low-frequency band signal) and the reconstruction parameter P (instead of the HFR parameter) to the first layer, the same architecture can be trained for the reconstruction of stereo audio signals. Where the output is the second monaural audio signal (instead of the high-frequency band audio signal), or the output is the second monaural audio signal (instead of the high-frequency band audio signal) and the third monaural audio signal (instead of the enhanced low-frequency band audio signal).

[0079] In FIG. 2b, a decoder 20 according to some embodiments is depicted. The decoder 20 includes some components that are the same as the corresponding components of the decoder in FIG. 2a (e.g., the mixing unit 26). The decoder 20 in FIG. 2b further includes an envelope estimator 22 and a flattening unit 23. Additionally, the decoder may include or be associated with a bitstream decoder as described in connection with the embodiment of FIG. 2a.

[0080] The envelope estimator 22 is configured to obtain a first monaural audio signal α and estimate the spectral envelope of the audio signal. In some embodiments, the spectral envelope is estimated for a number of frequency bands. For example, the spectral envelope is a parametric representation of the spectral energy of each QMF frequency band in the first monaural audio signal α. The spectral envelope can be represented by one, two, or at least three parameter values per frequency band. In some embodiments, the audio signal is represented by a predetermined number (e.g., 32) of QMF frequency bands that vary over time in segments, where each frequency band is associated with one reconstruction parameter (e.g., IID value, ICC value, or IPD value) updated for each time period.

[0081] The spectral envelope is provided to the flattening unit 23. The flattening unit 23 is configured to flatten the first monaural audio signal α, thereby providing a flattened sample α of the first monaural audio signal α to the neural network model 24b. F Accordingly, the neural network model 24b is trained to predict a flattened sample of the second monaural audio signal β* F when given the flattened sample α of the first monaural audio signal α. F Thus, while the neural network model 24a in FIG. 2a is trained to operate on the original (unflattened) samples of the first monaural audio signal α, the neural network model 24b is trained to operate on the flattened sample α. F The neural network model 24b also receives the reconstruction parameter P as an input.

[0082] Optionally, the neural network model 24b also receives the spectral envelope as an input, where the neural network model 24b is trained to predict the reconstructed second monaural audio signal β* based on the following three types of input data: the flattened sample of the first monaural audio signal α, the reconstruction parameter P, and the spectral envelope.

[0083] By allowing the neural network model 24b to operate in the flattened domain, the complexity of the neural network model 24b can be reduced (e.g., fewer layers and / or fewer parameters) and / or the training of the neural network model 24b can be made more efficient. Additionally, experiments have shown that providing the spectral envelope as a condition for the neural network model 24b is beneficial for prediction accuracy.

[0084] The reconstructed flattened sample β* F is provided to the inverse flattening unit 25, which performs the inverse operation of the flattening unit 23 to obtain the unflattened sample β*. F To this end, the spectral envelope and the reconstructed flattened sample β* F are provided to the inverse flattening unit 25 together. The inverse flattening unit 25 receives the flattened reconstructed second monaural audio signal β*. F(e.g., the flattened and reconstructed side audio signal) as input and outputs the de-flattened audio signal sample β* (i.e., the non-flattened and reconstructed audio signal).

[0085] The de-flattened and reconstructed second mono audio signal β* output by the de-flattening unit 25 is provided to the mixing unit 26, which mixes the de-flattened and reconstructed second mono audio signal β* with the first mono audio signal α to obtain the reconstructed left and right stereo audio signal pair L*, R*.

[0086] In FIG. 2c, a decoder 20 is schematically illustrated, where the neural network model 24c is trained to predict the (flattened) second reconstructed mono audio signal β* (F) and the (flattened) third reconstructed mono audio signal γ* (F) given the (flattened) first mono audio signal α (F) .

[0087] In some embodiments, the reconstructed third mono audio signal γ* is an enhanced version of the first mono audio signal α. For example, the first mono audio signal α is a mid audio signal, the reconstructed second mono audio signal β* is a side audio signal, and the reconstructed third mono audio signal γ* is an enhanced mid audio signal.

[0088] As described above, the first mono audio signal α can be compressed, quantized, or processed using any form of lossy audio coding technique. To this end, the neural network model 24c can be trained to produce an enhanced version of the first mono audio signal α in addition to the reconstructed second mono audio signal.

[0089] Optionally, the mixing unit 26 mixes the reconstructed third mono audio signal with the reconstructed second audio signal to produce the reconstructed left stereo audio signal L* and the reconstructed right stereo audio signal R* with enhanced quality.

[0090] FIG. 2d shows another embodiment of the decoder 20, where the neural network model 24d has been trained to directly output the samples of the (flattened) left and right stereo audio signal pair L* F , R* F given the samples of the first mono audio signal α and the reconstruction parameter P. This allows, for example, the mixing unit 26 to be completely omitted from the decoder 10.

[0091] The first mono audio signal α can be any one of a mid audio signal, a side audio signal, a left audio signal, and a right audio signal. In fact, there may be multiple variants, and generally speaking, the term "neural network models 24a, 24b, 24c, 24d" receives the first mono audio signal α as the first part of the first stereo format, and the reconstruction parameter P that describes the attributes of the second part of the first stereo format. The neural network models 24a, 24b, 24c, 24d are trained to output: (a) a reconstructed first format audio signal as the second part of the first stereo format; or (b) two reconstructed second format audio signals as the first part and the second part of the second stereo format, where the second stereo format is different from the first stereo format. For example, the neural network models 24a, 24b, 24c, 24d obtain the left audio signal and the reconstruction parameter associated with the right audio signal, and output the right audio signal, or the neural network models 24a, 24b, 24c, 24d obtain the mid audio signal and the reconstruction parameter associated with the side audio signal, and output a pair of left and right stereo audio signals. As a further example, the first mono audio signal α is a mid audio signal, and the parameter P describes the attributes of the corresponding side audio signal, where the neural network model 24d directly predicts a pair of reconstructed (flattened) left and right stereo audio signals L* F , R* F .

[0092] Although the decoder 20 in FIG. 2d operates on the flattened samples, it can be understood that the envelope estimator 22, the flattening unit 23, and the inverse flattening unit 25 can be omitted to allow the neural network module 24d to operate on the unflattened samples.

[0093] In the embodiments depicted in FIGS. 1a to 1b and FIGS. 2a to 2d above, it is illustrated that the encoder 10 receives the original left audio signal L and the original right audio signal R. In some embodiments (not shown), the encoder 10 instead receives the original audio signal in a mid-side format or any other type of stereo audio signal format. Regardless of the type of stereo format received by the encoder 10, the encoder 10 encodes a bitstream B that carries the representation of the mono audio signal and the reconstruction parameter that describes at least one attribute of the relationship between the original audio signals. For example, the encoder 10 receives the left audio signal L and the right audio signal R, and includes in the bitstream B one of the left audio signal L and the right audio signal R and the reconstruction parameter P that describes the attributes of the other of the left audio signal L and the right audio signal R.

[0094] The encoder 10 and decoder 20 described above can operate on audio signals in the time domain and / or in the frequency domain (e.g., in the QMF domain). For example, the encoder 10 converts a first mono audio signal and reconstruction parameters into a time-frequency domain format (such as QMF). Correspondingly, the neural network models 24a, 24b, 24c, 24d can be trained to predict a second (and optionally a third) mono audio signal based on the time-domain samples of the first mono audio and the reconstruction parameters describing the time-domain attributes. Alternatively, the neural network models 24a, 24b, 24c, 24d can be trained to predict a second (and optionally a third) mono audio signal based on the frequency-domain samples of the first mono audio and the reconstruction parameters describing the frequency-domain attributes.

[0095] In a traditional decoder without the neural network models 24a, 24b, 24c, 24d, other components in the decoder (e.g., the upmix unit) typically operate in a filter bank domain with a predetermined number of frequency bands. Then, the neural network models 24a, 24b, 24c, 24d can preferably be trained to operate in the same filter bank domain to facilitate easy implementation in a traditional decoder.

[0096] Figures 3a, 3b, 3c, 3d, and 3e depict some embodiments of the different neural network models described above and specific examples of the first mono audio signal, the second mono audio signal, and the third mono audio signal.

[0097] The neural network model 24a in Figure 3a is trained to obtain samples of a mid audio signal M and reconstruction parameters P indicating the attributes of the associated side audio signal S S , and output the reconstructed side audio signal S*. That is, the neural network model 24a is a single-output network, and the first mono audio signal and the second mono audio signal form a mid-side format stereo audio signal pair.

[0098] Except for the fact that the neural network model 24b in Figure 3b is trained to operate on flattened samples and optionally reconstruction parameters P associated with the flattened audio signal S , the neural network model 24b in Figure 3b is identical to the neural network model 24a in Figure 3a. In some embodiments, the spectral envelope (determined in conjunction with flattening the samples) is also provided as additional input data to the neural network model 24b. Experiments have shown that when the samples of the first mono audio signal (e.g., the mid audio signal M) are flattened, providing the envelope to the neural network model 24b helps the performance of the neural network model 24b.

[0099] The neural network model 24c of FIG. 3c also operates on the flattened audio signal samples. However, it is conceivable that the same neural network model 24c can also be trained to operate on non-flattened samples. The neural network model 24c is a dual-output network that is trained to obtain a flattened mid-audio signal M F (the first mono audio signal) and reconstruction parameters P associated with the corresponding side audio signal S , and outputs two flattened audio signals: the reconstructed side audio signal S* F (the second mono audio signal) and the enhanced reconstructed mid-audio signal M* F (the third mono audio signal). These audio signals S* F 、M* F form their own stereo audio signal pair, and can (after inverse flattening) be output as a stereo audio signal or be mixed to form different stereo audio signal pairs (e.g., left-right stereo audio signal pair). Additionally, as in the embodiment described in FIG. 3b, in some embodiments, the neural network model 24c is provided with a spectral envelope as additional input data.

[0100] The neural network model 24d of FIG. 3d also operates on the flattened audio signal samples. However, it is conceivable that the same neural network model 24d can also be trained to operate on non-flattened samples. Similar to the neural network model 24c, the neural network model 24d of FIG. 3d outputs two audio signals. However, the reconstructed audio signals L* F 、R* F output by the neural network model 24d have a different stereo format from the first mono audio signal M F . As seen in FIG. 3d, the first mono audio signal is the mid-audio signal in a mid-side stereo format, while the output reconstructed stereo audio signal is in a left-right stereo format.

[0101] Additionally, as in the embodiments described in FIGS. 3b and 3c, in some embodiments, the neural network model 24d is provided with a spectral envelope as additional input data.

[0102] Although FIGS. 3a to 3d depict many possible embodiments, these examples are not exhaustive. As seen in FIG. 3e, it is also conceivable that the first mono audio signal provided to the neural network model 24e is the left audio signal L or the right audio signal R, and the reconstruction parameters P L / RIndicates an attribute of the other of the left audio signal L or the right audio signal R. The output of the neural network model 24e can be a single signal or a dual signal. A single signal is a reconstruction of one of the left audio signal or the right audio signal that does not constitute the first mono audio signal. The dual output audio signals can be the reconstructed / enhanced left audio signal L and right audio signal R (i.e., the stereo format maintained by the neural network model), or a pair of reconstructed stereo audio signals in a different stereo format such as the mid-side format M, S, etc. Additionally, in some embodiments, the neural network model 24e is also provided with the spectral envelope representing the flattened samples of the left audio signal L or the right audio signal R, which may contribute to performance.

[0103] Thus, it is conceivable that the neural network 24e and the stereo decoder operate in a manner similar to the neural network and stereo decoder embodiments described above, where the input first mono audio signal is any one of the side audio signal S, the left audio signal L, and the right audio signal R.

[0104] Referring to the flowchart in FIG. 4a and FIG. 2a, a method for decoding a stereo audio signal will now be described in detail. In step S1, the decoder 20 receives a bitstream B that includes an encoded first mono audio signal α and reconstruction parameters P. In step S2, the decoder 20 decodes the encoded first mono audio signal α and the encoded reconstruction parameters P and provides the decoded first mono audio signal α together with the reconstruction parameters P to the neural network model 24a. In step S3a, the neural network model 24a reconstructs a second mono audio signal β*, and the first mono audio signal and the second mono audio signal form a pair of stereo audio signals. For example, the first mono audio signal α and the second mono audio signal β are the mid stereo audio signal and the side stereo audio signal or the left stereo audio signal and the right stereo audio signal. Optionally, in step S4a, the first mono audio signal and the second mono audio signal are provided to and mixed by the mixing unit 26 to convert the first mono audio signal and the second mono audio signal into a different alternative stereo format.

[0105] Embodiments can also be envisioned for reconstructing two audio signals, as will now be described with reference to FIGS. 2c and 4b, where the method further includes, in step S1, receiving a bitstream; and in step S2, decoding the first mono audio signal α, as described above. However, step S3b is also performed, where the neural network model 24b reconstructs a third mono audio signal γ* in addition to the second mono audio signal β* reconstructed in step S3a. Steps S3a and S3b can be performed sequentially or simultaneously. The reconstructed second mono audio signal β* and the reconstructed third mono audio signal γ* form a stereo audio signal pair, which has the same format as the format to which the first mono audio signal belongs, or a different stereo format that can be output by the decoder 20. However, it may be desirable to output stereo audio in a format different from that of the second mono audio signal β* and the third mono audio signal γ*, which means that these signals are optionally mixed in the mixing unit 26 in step S4b to create a stereo audio signal in the desired format.

[0106] In FIG. 5a, a setup for training any of the neural network models described above is depicted, and in FIG. 5b, a method for training any such neural network is illustrated. In step T1, training data is obtained, for example, from a database 30 that includes the training data. The training data includes at least one example of a stereo audio signal pair (e.g., a left-right stereo audio signal pair L, R), and it is provided to the stereo encoder 10. In step T2, the stereo encoder 10 encodes the stereo audio signal pair into a bitstream that contains a first mono audio signal (such as a mid audio signal M) and reconstruction parameters P that indicate at least one attribute of the stereo audio signal associated with the first mono audio signal.

[0107] The encoding process implemented by the encoder 10 may be a lossy process, which means that the information contained in the bitstream may not include enough information to perfectly reconstruct the stereo audio signal pair L, R input to the encoder 10.

[0108] In step T3, the first mono audio signal and the reconstruction parameters are provided to the decoder 20, which outputs the reconstructed stereo audio signal pair L*, R*. The decoder 20 can be any of the decoders described above in connection with FIGS. 2a, 2b, 2c, 2d and includes the neural network model 24 that is the object of training.

[0109] In step T4, the reconstructed stereo audio signal pair L*, R* is provided to a loss function unit 40, which compares the original stereo audio signal pair L, R with the reconstructed stereo audio signal pair L*, R* and determines a difference metric (also referred to as a loss) between the original and the reconstructed stereo audio signal pairs.

[0110] It is conceivable that a variety of different loss functions can be used to determine the difference metric. A suitable example of a loss function is the negative log-likelihood (NLL) loss. It is also conceivable that a discriminator (a so-called learnable loss function) can be used, where the decoder 20 (including the neural network model 24) acts as a generator and the loss function unit 40 includes an additional neural network acting as a discriminator. Subsequently, the discriminator and the generator are trained using a conventional generator / discriminator training.

[0111] Based on the difference metric determined by the loss function unit 40, in step T5, the internal weights and parameters of the neural network model 24 are updated to reduce the difference metric.

[0112] Subsequently, the above process is repeated using new training data (or at least augmented training data) used in each iteration until the weights and parameters of the neural network model 24 have been calibrated to produce a sufficiently small difference metric in various training data.

[0113] In FIG. 5a, it is assumed that the audio signal of the training data and the audio signal output by the decoder 20 are in the same left L - right R stereo format. However, in some embodiments, the training data is in a different stereo format (e.g., mid - side format), and / or there is a mismatch between the stereo format output by the decoder 20 and the format of the training data in the database 30. In such embodiments, the loss function unit 40 is configured to convert the stereo format output by the decoder 20 and / or the stereo format of the ground truth training data 30 in order to calculate the loss.

[0114] Figure 6 Another embodiment of the decoder 20 implementing the neural network module 24 is shown. The neural network module 24 can be any one of the neural network modules 24a, 24b, 24c, 24d, 24e described above, or a variant thereof that operates on flattened or unflattened audio signal samples.

[0115] Figure 6The decoder 20 therein includes a content analyzer 27 configured to determine a relevance level of the audio content of a bitstream (represented by the first mono audio signal and the reconstruction parameters). This relevance level indicates the level of correlation between the first mono audio signal α and the stereo audio signal associated with the first mono audio signal α. For example, if the bitstream carrying the first mono audio signal α and the reconstruction parameters P is obtained by encoding a pair of left and right audio signals, the relevance level will indicate the level of correlation between the left audio signal and the right audio signal.

[0116] In some embodiments, the relevance level is determined by the content analyzer based on the reconstruction parameters P. For example, the reconstruction parameters may include a parameter P that directly indicates the relevance level. In some embodiments, the content analyzer 27 includes a neural network trained to predict the relevance level based on samples of the first mono audio signal α and optionally also the reconstruction parameters P. For example, the content analyzer 27 may be trained to determine the type of audio content present in the first mono audio signal α (e.g., speech, music, applause, or rain), where some content types (e.g., applause or rain) are associated with a low relevance level.

[0117] The relevance level is provided to a selection module 28 that selects whether to provide the first mono audio signal M and the reconstruction parameters to a neural network module or a predetermined linear time-invariant (LTI) filter 29. The LTI filter 29 can be implemented using a delay line that applies an (optionally frequency-varying) delay to the audio signal in the time domain. Alternatively or additionally, the LTI filter 29 includes an infinite impulse response (IIR) and / or finite impulse response (FIR) filter that simulates reverberation.

[0118] The delay in the time domain can vary with frequency, e.g., higher frequency bands are subject to less delay. If the relevance level is below a predetermined threshold, the selection module 28 selects the neural network module 24; and if the relevance level is above the predetermined threshold, the selection module 28 selects the LTI filter 29. As described above, when the correlation between the channels of the stereo audio signal pair described by the first mono audio signal α and the reconstruction parameters P is low, the neural network model 24 is particularly suitable for reconstructing the second (and optionally the third) mono audio signal. Thus, in Figure 6 the decoder 20, the neural network model 24 is used when most needed, while the simpler LTI filter 29 is used for reconstructing the correlated stereo audio signal pair.

[0119] The LTI filter 29 refers to a filter including a set of filter coefficients determined through analysis for performing a specific task, which in this example is the task of imposing a time-domain delay and / or reverb on an audio signal. Additionally or alternatively, the delay filter includes a delay unit configured to introduce a predetermined (possibly frequency-varying) delay. Depending on the desired time delay (e.g., 10 milliseconds or 2 milliseconds), the filter coefficients of the LTI filter 29 can be analytically calculated. For example, if the LTI filter 29 introduces a delay that linearly decreases with frequency, the LTI filter will produce the impulse response of a chirp signal. Examples of the LTI filter 29 are presented, for example, in “Synthetic Ambience in Parametric StereoCoding” by [authors], 2004. When using the LTI filter, the reconstruction parameter P is provided to the mixing unit 26, which performs mixing to change the stereo format according to the reconstruction parameter P. This type of mixing taking into account the reconstruction parameter P is also described in detail in the above reference.

[0120] In some embodiments, the neural network model 24 includes at least one of the following: a plurality of (learnable) neural network layers, a non-linear activation layer (e.g., having a rectified linear unit ReLU), at least one long short-term memory (LSTM) layer, at least one recurrent layer (e.g., a layer including a gated recurrent unit GRU), which means that the neural network model 24 is significantly different from the LTI filter 29 that is both linear and time-invariant.

[0121] Unless otherwise specifically stated, it should be understood from the following discussion that throughout the discussion of the present disclosure, the use of terms such as “processing”, “computing”, “calculating”, “determining”, “analyzing”, etc. to refer to actions and / or processes of a computer hardware or a computing system or a similar electronic computing device that manipulate and / or transform data represented as physical (such as electronic) quantities into other data similarly represented as physical quantities.

[0122] It should be understood that in the above description of the exemplary embodiments of the present invention, various features of the present invention are sometimes combined in a single embodiment, figure, or description thereof in order to simplify the disclosure and assist in understanding one or more of the inventive aspects. However, the methods of the present disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. On the contrary, as reflected in the following claims, the inventive aspects lie in less than all the features of a single foregoing disclosed embodiment. Accordingly, the claims following the detailed description are hereby expressly incorporated into the detailed description, where each claim stands on its own as a separate embodiment of the present invention. In addition, although some of the embodiments described herein include some features included in other embodiments and not other features included in other embodiments, as will be understood by those skilled in the art, combinations of features of different embodiments are intended to be within the scope of the present invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0123] In addition, some embodiments herein describe methods or combinations of method elements that can be implemented by a processor of a computer system or by other devices that perform functions. Therefore, a processor having instructions for performing such a method or method element forms a means for performing the method or method element. It should be noted that when the method includes multiple elements (e.g., several steps), no order of these elements is implied unless otherwise specifically stated. In addition, the elements of the apparatus embodiments described herein are examples of apparatuses that perform the functions performed by the elements in order to implement the embodiments of the present invention. In the description provided herein, numerous specific details are set forth. However, it should be understood that embodiments of the present invention may be practiced without these specific details. In other instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0124] Accordingly, although specific embodiments of the present invention have been described, those skilled in the art will recognize that other and further modifications can be made thereto without departing from the spirit of the invention, and it is intended to claim all such changes and modifications that fall within the scope of the present invention. For example, while most of the depicted embodiments relate to a decoder configured to obtain a mid-stereo audio signal and reconstruction parameters indicating attributes of corresponding side audio signals to reconstruct left and right audio signals, it will be understood that many alternative decoders fall within the scope of the present disclosure. Such as a decoder configured to obtain side audio signals (and reconstruction parameters associated with the mid-audio signal) to reconstruct left and right audio signals, a decoder configured to obtain a left audio signal (and reconstruction parameters associated with the right audio signal) to reconstruct left and right audio signals, and a decoder configured to obtain a right audio signal (and reconstruction parameters associated with the left audio signal) to reconstruct left and right audio signals. Each of these exemplary decoders may also be configured to output a pair of stereo audio signals in a format different from a left-right format such as a mid-side format or a target mid-side format. Additionally, decoders with and without spectral flattening and associated components (envelope estimators, flattening units, inverse flattening units) are also contemplated.

Claims

1. A method for reconstructing a stereo audio signal, the method comprising: Receiving (S1) a bitstream comprising an encoded first mono audio signal and a set of reconstruction parameters; Decoding (S2) the encoded first mono audio signal to provide a first mono audio signal; Reconstructing (S3a) a second mono audio signal using a neural network system (24, 24a, 24b, 24c, 24d, 24e), the neural network system being trained to predict samples of the second mono audio signal given samples of the first mono audio signal and the reconstruction parameters; and wherein the first mono audio signal and the reconstructed second mono audio signal form a stereo audio signal pair.

2. A method for reconstructing a stereo audio signal, the method comprising: Receiving (S1) a bitstream comprising an encoded first mono audio signal and a set of reconstruction parameters; Decoding (S2) the encoded first mono audio signal to provide a first mono audio signal; Reconstructing (S3a, S3b) a second mono audio signal and a third mono audio signal using a neural network system (24c, 24d, 24e), the neural network system being trained to predict samples of the second mono audio signal and samples of the third mono audio signal given samples of the first mono audio signal and the reconstruction parameters, wherein the reconstructed second mono audio signal and the reconstructed third mono audio signal form a stereo audio signal pair.

3. The method according to claim 1 or claim 2, wherein The first mono audio signal is a mid audio signal.

4. The method according to claim 3, wherein The reconstructed second mono audio signal is a side mono audio signal.

5. The method according to claim 2, wherein The reconstructed second mono audio signal is a side mono audio signal, and wherein the reconstructed third mono audio signal is an enhanced mid audio signal.

6. The method according to claim 2 or claim 3, wherein, The reconstructed second mono audio signal is a left mono audio signal, and the reconstructed third mono audio signal is a right mono audio signal.

7. The method according to any one of the preceding claims, wherein, The reconstruction parameters indicate attributes of the stereo audio signal associated with the first mono audio signal.

8. The method according to claim 7, wherein, The reconstruction parameters are at least one of the following: Inter-channel intensity difference IID parameter, Inter-channel cross-correlation ICC parameter, and Inter-channel phase difference IPD parameter.

9. The method according to any one of the preceding claims, wherein, The neural network system (24, 24a, 24b, 24c, 24d, 24e) is trained to operate on flattened audio signal samples, the method further comprising: Performing envelope flattening on the first mono audio signal to produce a flattened first mono audio signal; and Providing the flattened first mono audio signal to the neural network system (24, 24a, 24b, 24c, 24d, 24e); and Performing inverse flattening on at least the reconstructed second mono audio signal.

10. The method according to claim 9, further comprising: Determining the envelope of the first mono audio signal, Wherein, the envelope flattening and inverse flattening are based on the envelope determined from the first mono audio signal.

11. The method according to any one of the preceding claims, wherein, The neural network system (24, 24a, 24b, 24c, 24d, 24e) operates on samples in the time domain, frequency domain, or filter bank domain.

12. The method according to any one of the preceding claims, further comprising: Determining a correlation level of at least a portion of the first mono audio signal, the correlation level indicating a level of correlation between the first mono audio signal and a stereo audio signal associated with the first mono audio signal; If the correlation level is below a predetermined threshold level, providing the portion of the first mono audio signal to the neural network model (24, 24a, 24b, 24c, 24d, 24e); Otherwise, processing the portion of the first mono audio signal using a linear time-invariant (LTI) filter (29) to obtain an alternative reconstructed second mono audio signal, the LTI filter (29) comprising a set of analytically determined filter coefficients.

13. A computer program product comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 12.

14. A computer-readable storage medium storing the computer program according to claim 13.

15. A decoder, the decoder comprising a processor and a memory coupled to the processor, wherein, The processor is adapted to perform the steps of the method according to any one of claims 1 to 12.

16. A method for training a neural network model (24, 24a, 24b, 24c, 24d, 24e) for stereo reconstruction, the method comprising: Obtaining training data (T1), the training data (30) comprising stereo audio signal pairs; Encoding (T2) the stereo audio signal pairs into an encoded stereo audio signal, the encoded stereo audio signal comprising a first mono audio signal and reconstruction parameters; Reconstructing (T3) a second mono audio signal using a neural network system (24, 24a, 24b, 24c, 24d, 24e) that is trained to predict samples of the second mono audio signal given samples of the first mono audio signal and the reconstruction parameters; Determining (T4) a difference metric between the reconstructed second mono audio signal and a ground truth second mono audio signal associated with the stereo audio signal pairs of the training data; And Modifying (T5) internal weights of the neural network model (24, 24a, 24b, 24c, 24d, 24e) based on the determined difference.

17. The method according to claim 16, wherein, The neural network model (24c, 24d, 24e) is further configured to predict samples of a third mono audio signal given samples of the first mono audio signal and the reconstruction parameters, the method further comprising: Reconstructing the third mono audio signal using the neural network model (24c, 24d, 24e); and Determine (T5) the difference metric between the reconstructed third monaural audio signal and the ground truth third monaural audio signal associated with the stereo audio signal pair of the training data.

Images