Spatial Noise Filling in Multi-Channel Codecs

JP2024503186A5Active Publication Date: 2024-12-06DOLBY LABORATORIES LICENSING CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023532192
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-24
Filing Date
2021-12-01
Publication Date
2024-12-06
Estimated Expiration
2041-12-01

AI Technical Summary

Technical Problem

Multichannel codecs experience noise ambience collapse at low bit rates due to limited bits for encoding, leading to coding artifacts and reduced background noise energy, especially at high frequencies, which affects the perceived ambience of audio signals.

Method used

Implement spatial noise filling techniques in multichannel codecs to generate multichannel noise with desired spatial and spectral shapes, using encoder-side information and decoder-side characteristics to recreate background noise ambience and fill spatial holes.

Benefits of technology

Enhances the perceived ambience of multichannel audio signals by improving the reproduction of background noise, addressing noise ambience collapse at low bit rates.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

An embodiment of spatial noise filling in a multi-channel codec is disclosed. In one embodiment, a method for recreating a background noise ambience in a multi-channel codec by generating spatial hole-filling noise includes: calculating a noise estimate based on main downmix channels generated from an input audio signal representing a spatial audio scene with a background noise ambience; calculating spectral shaping filter coefficients based on the noise estimate; spectrally shaping the multi-channel noise signal using the spectral shaping filter coefficients and a noise distribution, the spectral shaping resulting in a diffuse multi-channel noise signal with uncorrelated channels; spatially shaping the diffuse uncorrelated multi-channel noise signal with uncorrelated channels based on the noise ambience of the spatial audio scene; and adding the spatially and spectrally shaped multi-channel noise signal to a multi-channel codec output to synthesize the background noise ambience of the spatial audio scene.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 120,658, filed December 2, 2020, and U.S. Provisional Application No. 63 / 283,187, filed November 24, 2021, all of which are incorporated by reference in their entireties.

[0002] Technical Field This disclosure relates generally to audio processing in immersive voice and audio contexts. [Background technology]

[0003] Speech and audio encoder / decoder ("codec") standards development has recently focused on the development of multi-channel codecs for immersive voice and audio services (IVAS). IVAS is expected to support a wide range of audio service functions, including but not limited to mono-to-stereo upmixing and fully immersive audio encoding, decoding, and rendering. IVAS is intended to be supported by a wide range of devices, endpoints, and network nodes, including but not limited to mobile and smart phones, electronic tablets, personal computers, conference phones, conference rooms, virtual reality (VR) and augmented reality (AR) devices, home theater devices, and other suitable devices. These devices, endpoints, and network nodes may have a variety of acoustic interfaces for sound capture and rendering. The ability of a multi-channel codec to reproduce the encoder input audio scene at the decoder output depends on the number of downmix channels that are encoded, the coding artifacts introduced by the mono codec, whether the decorrelator used in the decoder is able to output downmix channels that are uncorrelated with respect to the primary downmix channel, and the accuracy of the side information that is encoded. At low bitrates due to bit scarcity, there is often a trade-off between preserving the audio essence and the background noise ambience of the input scene: preserving the audio essence is perceptually more important, which leads to the crushing of the background noise ambience. Summary of the Invention [Means for solving the problem]

[0004] An embodiment of spatial noise filling in a multi-channel codec is disclosed. In an embodiment, spatial noise filling includes: generating multi-channel noise with a desired spatial and spectral shape with minimal or no additional information from the encoder; and adding said multi-channel noise to the final upmixed output at the decoder to recreate background noise ambience and fill spatial holes. The spectral shape of the multi-channel noise is determined by the dominant downmix channel, which is, for example, a representation of the W channel for a first-order Ambisonics (FoA) input signal format and a representation of the Mid channel for a Mid-Side (M / S) input signal format. The spatial shape of the multi-channel noise is determined by spatial information from the input spatial audio scene. This spatial information can be extracted from the side information (extracted spatial metadata) sent by the encoder, or from the spatial characteristics of the upmix output at the decoder, or both. In one embodiment, the spatial shape of the multi-channel noise is extracted from both the side information (spatial metadata) transmitted by the encoder and the spatial characteristics of the upmixed output at the decoder.

[0005] Other embodiments disclosed herein are directed to systems, devices, and computer readable media. Details of the disclosed embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims.

[0006] Certain embodiments disclosed herein provide one or more of the following advantages: The disclosed spatial noise filling techniques address the problem of noise ambience collapse at low bit rates in multi-channel codecs by improving the perceived ambience of a multi-channel audio signal. [Brief description of the drawings]

[0007] The figures show a particular arrangement and order of schematic elements, such as those representing devices, units, instruction blocks, data elements, etc., for ease of description. However, those skilled in the art should understand that the particular order and arrangement of schematic elements in the figures does not imply that a particular order or sequence of processing, or separation of processes, is required. Moreover, the inclusion of a schematic element in a drawing is not intended to imply that such element is required in all embodiments, or that features represented by such element cannot be included in or combined with other elements in some embodiments.

[0008] Furthermore, in drawings where connecting elements such as solid or dashed lines or arrows are used to indicate a connection, relationship, or association between two or more other schematic elements, the absence of such connecting elements is not intended to mean that the connection, relationship, or association cannot exist. That is, some connections, relationships, or associations between elements are not shown in the drawings so as not to obscure the disclosure. Furthermore, for ease of explanation, a single connecting element is used to represent multiple connections, relationships, or associations between elements. For example, where a connecting element represents communication of signals, data, or instructions, one skilled in the art should understand that such element represents one or more signal paths as necessary to affect the communication.

[0009] [Figure 1] 1 illustrates a use case of an IVAS system according to an embodiment.

[0010] [Diagram 2] 1 is a block diagram of a multi-channel codec according to an embodiment.

[0011] [Diagram 3] FIG. 2 is a block diagram of a decoder for processing a 1-channel downmix signal using spatial noise filling according to an embodiment.

[0012] [Figure 4] FIG. 2 is a block diagram of a decoder for processing a 1-channel downmix signal using spatial noise filling with noise spectral shaping according to an embodiment.

[0013] [Diagram 5] FIG. 2 is a flow diagram of a process for recreating background noise ambience in a multi-channel codec by generating spatial hole-filling noise, according to an embodiment.

[0014] [Figure 6] FIG. 6 is a block diagram of an exemplary device architecture for implementing the features and processes described with reference to FIGS. 1 to 5, according to one embodiment.

[0015] The use of the same reference symbols in the various drawings indicates similar elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. It will be apparent to one of ordinary skill in the art that the various embodiments described may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments. The following describes several features that can be used independently of each other or in any combination with other features.

[0017] notation As used in this application, the term "comprises" and variations thereof should be read as open-ended terms meaning "including, but not limited to." The term "or" should be read as "and / or" unless the context clearly indicates otherwise. "Based on" should be read as "based at least in part on." "One embodiment" and "an embodiment" should be read as "at least one embodiment." "Another embodiment" should be read as "at least one other embodiment." The terms "determined," "determining," or "determining" should be read as obtaining, receiving, calculating, calculating, estimating, predicting, or deriving. Moreover, in the following description and claims, unless otherwise defined, all scientific and technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0018] Examples of IVAS use cases FIG. 1 illustrates use cases of an IVAS system 100 according to an embodiment. In some embodiments, various devices communicate through a call server 102 configured to receive audio signals from a public switched telephone network (PSTN) or public land mobile network devices (PLMNs), for example as indicated by PSTN / other PLMN 104. Use cases support legacy devices 106 that render and capture audio in mono only, including but not limited to devices that support enhanced voice services (EVS), multi-rate wideband (AMR-WB) and adaptive multirate narrowband (AMR-NB). Use cases also support user equipment (UE) 108, 114 that captures and renders stereo audio signals, or UE 110 that captures and binaurally renders mono signals into a multi-channel signal. Use cases also support immersive stereo signals that are captured and rendered by video conferencing room systems 116, 118, respectively. Use cases also support the computer 112 for stereo capture and immersive rendering of stereo audio signals for a home theater system 120, as well as mono capture and immersive rendering of audio signals for virtual reality (VR) gear 122 and immersive content ingestion 124.

[0019] Example IVAS Codec 2 is a block diagram of an IVAS codec 200 for encoding and decoding an IVAS bitstream according to an embodiment. The IVAS codec 200 includes an encoder and a far-end decoder. The IVAS encoder includes a spatial analysis and downmix unit 202, a quantization and entropy coding unit 203, a core encoding unit 206 (e.g., an EVS encoding unit) and a mode / bitrate control unit 207. The IVAS decoder includes a quantization and entropy decoding unit 204, a core decoding unit 208 (e.g., an EVS decoding unit), a spatial synthesis / rendering unit 209 and a decorrelator unit 211.

[0020] The spatial analysis and downmix unit 202 receives an N-channel input audio signal 201 representing an audio scene. The input audio signal 201 includes, but is not limited to: a mono signal, a stereo signal, a binaural signal, a spatial audio signal (e.g. multi-channel spatial audio objects), FoA, Higher Order Ambisonics (HoA), and any other audio data. The N-channel input audio signal 201 is downmixed by the spatial analysis and downmix unit 202 to a specified number of downmix channels (N_dmx), where in this example N_dmx is <=N. The spatial analysis and downmix unit 202 also generates side information (e.g. spatial metadata), which can be used by the far-end IVAS decoder to synthesize the N-channel input audio signal 201 from the N_dmx downmix channels, the spatial metadata and the decorrelated signal generated in the decoder. In some embodiments, the spatial analysis and downmix unit 202 implements complex advanced coupling (CACPL) for analyzing / downmixing stereo / FoA audio signals and / or a spatial reconstructor (SPAR) for analyzing / downmixing FoA audio signals. In other embodiments, the spatial analysis and downmix unit 202 implements other formats.

[0021] The N_dmx channels are encoded by N_dmx instances of a mono codec contained in the core encoding unit 206, and the side information (e.g., spatial metadata (MD)) is quantized and encoded by the quantization and entropy encoding unit 203. The coded bits are then packed together into a bitstream(s) and transmitted to the IVAS decoder. In the embodiment shown, an exemplary embodiment of the underlying codec is EVS, although any suitable mono, stereo or multi-channel codec can be used to generate the encoded bitstream.

[0022] In some embodiments, several increasingly coarse quantization levels may be included (e.g., fine, medium, coarse, very coarse quantization), and the entropy coding may include Huffman coding or arithmetic coding.

[0023] In some embodiments, the core encoding unit 206 is a 3GPP TS 26.445 compliant EVS encoding unit 206 providing a wide range of functionality, such as improved quality and coding efficiency for narrowband (EVS-NB) and wideband (EVS-WB) speech services, improved quality using super-wideband (EVS-SWB) speech, improved quality for mixed content and music in conversational applications, robustness to packet loss and delay jitter, and backward compatibility with the AMR-WB codec.

[0024] In some embodiments, the EVS encoding unit 206 includes a pre-processing and mode / bitrate control unit 207, which selects between a speech encoder for encoding the speech signal and a perceptual encoder for encoding the audio signal at a specified bitrate based on the output of the mode / bitrate control unit 207. In some embodiments, the speech encoder is an improved version of Algebraic Code Excited Linear Prediction (ACELP) and is extended with specialized Linear Prediction (LP) based modes for various speech classes. In some embodiments, the perceptual encoder is a Modified Discrete Cosine Transform (MDCT) encoder with improved efficiency at low delay / low bitrate, designed to perform seamless and reliable switching between the speech encoder and the audio encoder.

[0025] At the decoder, the N_dmx channels are decoded by corresponding N_dmx instances of a mono codec in the core decode unit 208, and the side information is decoded by the quantization and entropy decoding unit 204. The primary downmix channel (e.g. the W channel in the FoA signal format) is fed to a decorrelator unit 211, which generates N-N_dmx decorrelated channels. The N_dmx downmix channels, the N-N_dmx decorrelated channels and the side information are fed to a spatial synthesis / rendering unit 209, which uses these inputs to synthesize or recreate the original N-channel input audio signal. In one embodiment, the N_dmx channels are decoded by a mono codec other than EVS. In other embodiments, the N_dmx channels are decoded by a combination of one or more multi-channel core coding units and one or more single-channel core coding units.

[0026] Multi-channel codecs such as the IVAS codec 200 suffer from noise ambience collapse (hereinafter also referred to as "spatial holes") at low bit rates. At low bit rates, the number of downmix channels is usually very small (e.g., N_dmx=1 downmix channel) and the number of bits available for the mono codec to code the downmix channels is also small. This leads to coding artifacts and lowers the overall energy of the background noise, especially at the high frequencies that form the ambience. Fewer downmix channels also means that the decorrelator needs to generate more uncorrelated channels. Typically, the decorrelator cannot generate completely uncorrelated channels with the desired spectral shape. Finally, the side information may be coarsely quantized due to the available bit budget. These problems lead to noise ambience collapse or spatial holes, which are solved by modifying the IVAS decoder to implement spatial noise filling, as described below with reference to Figures 3 and 4.

[0027] 3 is a block diagram of an IVAS decoder with a one-channel downmix signal and spatial noise filling, according to an embodiment. The exemplary IVAS decoder shown is a SPAR decoder 300 operating in one-channel downmix mode (N_dmx=1), which is configured to decode an encoded IVAS bitstream. It should be noted that the spatial noise filling techniques described below can be applied to any downmix configuration with any number of downmix signals.

[0028] The SPAR decoder 300 includes a bit unpacking unit 301, a core decoding unit 302 (core decoding unit 208 in FIG. 2), a noise estimation and spectral shaping parameter extraction unit 303, a noise upmixer unit 304, a multi-channel noise spatial shaping unit 305, a spatial metadata (MD) decoding unit 306 (quantization and entropy decoding unit 204 in FIG. 2), a decorrelation unit 307 (decorrelation unit 211 in FIG. 2), an upmix unit 308 (spatial synthesis / rendering unit 209 in FIG. 2), and a spatial noise addition unit 309.

[0029] The bit unpack unit 301 receives the encoded IVAS bitstream(s) generated upstream by the IVAS encoder. The IVAS bitstream includes quantized and encoded spatial metadata (MD) and encoded core coder bits. The bit unpack unit 301 unpacks the IVAS bitstream and sends the MD bits to the MD decode unit 306 and the core coded bits to the core decode unit 302. In a one-channel downmix configuration for FoA, the core coded bits include only W' (representation of the W channel) coded bits.

[0030] The core decoding unit 302 decodes the core coded bits and generates active W' pulse code modulated (PCM) output data, which is fed to the noise estimation and spectral shaping parameter extraction unit 303 and the decorrelation unit 307. The noise estimation and spectral shaping parameter extraction unit 303 reads the VAD (Voice Activity Detector) / SAD (Speech Activity Detector) decision flag(s) in the bitstream metadata and extracts the spectral shaping parameters of the background noise when only background noise is present (VAD / SAD decision is 0). Note that when the VAD / SAD decision is 1, the spectral shaping parameters are static. In other embodiments, the bits received by the block 302 may be coded by a different core codec other than EVS, and thus the block 302 can be a different core codec other than EVS.

[0031] The spectral parameters are fed to a noise upmixer unit 304 which generates N uncorrelated noise channels (e.g., N=4 for FoA encoding) with the same spectral shape as the background noise in the W′ channel. In one embodiment, these noise channels are generated based on a Gaussian white noise distribution with a different seed for each of the N channels, thereby generating completely uncorrelated noise channels.

[0032] Once the spectral shaping parameters are extracted, the noise upmixer unit 304 generates multi-channel uncorrelated noise regardless of the VAD / SAD decision value. The output of the noise upmixer unit 304 is fed to the multichannel noise spatial shaping unit 305, which spatially shapes the N uncorrelated noise channels based on the spatial metadata output by the MD decoding unit 306 and / or the spatial parameters extracted from the output of the upmix unit 308 (upmixed SPAR FoA output without spatial noise filling). Although the spatial parameters of the background noise modeling are calculated only during inactive frames (e.g., when only background noise is present, i.e., when the VAD / SAD decision is 0), the multichannel noise spatial shaping unit 305 generates spatial noise regardless of whether the current frame is active or inactive (e.g., VAD / SAD decision is 0 or 1). This is done by freezing the spatial parameters calculated in the last inactive frame during the active frame.The MD bits output by the Bit Unpack Unit 301 are fed to an MD Decode Unit 306, which decodes the spatial metadata encoded by an IVAS encoder (not shown).

[0033] The output of the core decode unit 302 is also fed to a decorrelation unit 307, which generates three decorrelated outputs 307 (decorrelated with respect to the W' channel of the downmix). The outputs of the decorrelation unit 307 and the MD decode unit 306 are fed to an upmix unit 308, which generates FoA output channels from the downmix channels, the decorrelated channels output by the decorrelation unit 307 and the spatial metadata MD. At high bitrates, the output of the upmix unit 308 resembles the FoA input to the SPAR encoder, but at low and mid-range bitrates, the output of the upmix unit 308 may suffer from ambience collapse.

[0034] To prevent ambience collapse, the spatial noise addition unit 309 adds spatially and spectrally shaped multi-channel noise with desired spatial and spectral shapes to the output of the upmix unit 308. In some embodiments, the spatial noise addition unit 309 adds multi-channel noise with desired spatial and spectral shapes to the parametrically generated channels at the output of the upmix unit 308. In one-channel downmix mode, the Y, X, and Z channels are parametrically generated by the SPAR 300 decoder using the spatial metadata sent from the SPAR encoder, the main downmix channel (W′ downmix channel), and the output of the decorrelation unit 307, so that the masking noise is added only to the Y, X, and Z channels. In a two-channel downmix mode, the X and Z channels are generated parametrically by the SPAR decoder 300 using the spatial metadata transmitted from the SPAR encoder, the downmix channels, and the output of the decorrelation unit 307, so that masking noise is added only to the X and Z channels. In a three-channel downmix mode, the Z channel is generated parametrically by the SPAR decoder 300 using the spatial metadata transmitted from the SPAR encoder, the downmix channels, and the output of the decorrelation unit 307, so that masking noise is added only to the Z channel.

[0035] In one embodiment, the Noise Upmixer unit 304 generates four uncorrelated masking noise channels with the same spectral shape as the background noise in the W' channel, applies a low-order high-pass filter to limit the effect of the spatial masking noise to high frequencies (as ambience noise crushing is typically more perceived at higher frequencies), and then applies a smoothing gain to further smooth the effect of the spatial masking noise.

[0036] In one embodiment, the multi-channel noise spatial shaping unit 305 checks the VAD / SAD decision value in the EVS bitstream metadata, takes the output of the upmix unit 308, and passes it through a high-pass filter to emphasize higher frequencies more. The high-pass filtered output is then used to calculate covariance estimates between all four channels. The covariance estimates are used to generate spatial parameters that are used to spatially shape the fully diffuse (uncorrelated) masking noise. In one embodiment, the covariance estimates are broadband covariance estimates, and the spatial parameters are SPAR spatial parameters (e.g., prediction coefficients and correlation coefficients). The masking noise shaping parameters are calculated only when background noise is present (e.g., the VAD / SAD decision is 0) and are otherwise static when speech or audio is present in the input audio signal (e.g., the VAD / SAD decision is 1).

[0037] In one embodiment, the multi-channel noise spatial shaping unit 305 checks the VAD / SAD decision output and spatially shapes the output of the noise upmixer unit 304 using the decoded spatial MD generated by the MD decoding unit 306. In one embodiment, the spatial MD output of the MD decoding unit 306 is further smoothed and recalculated to emphasize more higher frequencies (e.g., high-pass filtered) before being applied to the output of the noise upmixer unit 304. The multi-channel noise spatial shaping parameters are calculated only when background noise is present (e.g., the VAD / SAD decision is 0) and are static when speech or sound is detected (e.g., the VAD / SAD decision is 1).

[0038] In an embodiment, the spatial noise addition unit 309 adds multi-channel noise with the desired spatial and spectral shape only to the parametrically generated channels at the multi-channel decoder output. In an embodiment, the spatial noise filling can be done with any multi-channel codec other than IVAS or SPAR for an N-channel multi-channel input (N>=1). The same spatial noise filling approach can be applied, where the multi-channel noise is spectrally shaped based on the main channels and the spatial shape of the multi-channel noise is determined by either or both of the spatial metadata sent by the encoder and the synthesized multi-channel output. Then, at the decoder, the multi-channel noise with the desired spectral and spatial shape can be added to the synthesized multi-channel output.

[0039] 4 is a block diagram of a SPAR decoder 400 operating in a one-channel downmix configuration according to an embodiment, where spatial noise filling uses an internal module of the core codec to extract the spectral characteristics of the background noise in the downmix channel. The following description of the further embodiment focuses on the differences between it and the aforementioned embodiment. Therefore, in the following description, features common to both embodiments may be omitted, and thus, unless otherwise required in the following description, it should be assumed that in the further embodiment, the features of the aforementioned embodiment are implemented, or at least can be implemented.

[0040] The SPAR decoder 400 includes a core decoder 409 and an MD decoder and upmixer 410. The core decoder 409 includes a core decode unit 401, a noise estimation unit 402, a noise upmixer unit 403, and a single channel noise filling unit 404, which is already present in the core decoder 409 and adds spectrally shaped noise to the decoded output to mask core coding artifacts. The MD decoder and upmixer 410 includes a decorrelation unit 405, an upmix unit 407, and a spatial shaping and noise filling unit 408.

[0041] In one embodiment, the noise spectral shaping is implemented within the core decoder 409 using a spectral shaping module therein. Note that the noise estimation and spectral shaping parameter extraction unit 303 and a section of the noise upmixer unit 304 in the SPAR decoder 300 shown in Figure 3 are also present within the core decode unit 302 (units 402 and 403).

[0042] Note that the noise estimation and spectral shaping parameter extraction unit 303 in the SPAR decoder 300 shown in FIG. 3 is also present inside the core decode unit 302 (unit 402). The core decode unit 302 also has a single-channel noise generator that uses a Gaussian white noise distribution as an excitation signal and spectrally shapes it according to the spectral parameters generated by the noise estimation unit 402. This single-channel noise generator can be easily modified into a multi-channel noise generator that generates multiple uncorrelated noise channels with the same spectral shape by using different seeds per channel for the Gaussian white noise distribution. This multi-channel noise generator is shown as unit 403 in FIG. 4 and is equivalent to unit 304 in FIG. 3.

[0043] In this embodiment, the decoder 409 decodes the representation of the W channel and the noise estimation unit 402 estimates the noise in the decoded data, which is used by unit 403 to generate four uncorrelated noise channels with the same spectral shaping. These noise channels are generated based on a Gaussian white noise distribution with a different seed for each channel, thus generating completely uncorrelated noise channels.

[0044] The SPAR decoder described above with reference to Figures 3 and 4 converts the FoA input audio signal, which represents an audio scene, into a set of downmix channels and spatial parameters that are used to regenerate the input signal in the SPAR decoder. The downmix signal can vary from 1 to 4 channels, and the parameters include a prediction parameter PR, a cross-prediction parameter C, and a decorrelation parameter P. These parameters are calculated from the covariance matrix of the windowed input audio signal and are calculated in a specified number of frequency bands (e.g. 12 frequency bands).

[0045] An exemplary representation of the SPAR parameter extraction is as follows:

[0046] 1. Predict all side signals (Y,Z,X) from the main audio signal W using equation [1]:

number

number

[0047] The above downmix is ​​also called passive W downmix, where W remains unchanged during the downmix process. Another aspect of downmix is ​​active W downmix, which allows some mixing of Y, X, Z channels into the W channel, as follows:

number

[0048] 2. Remix the W channel and the predicted (Y',Z',X') channels from acoustically most significant to least significant as shown in equation [4], where remixing involves reordering or recombining the channels based on some methodology.

number

[0049] One embodiment of the remix may be to reorder the input channels W,Y',X',Z', given the assumption that audio cues from left and right are more important than front or back, and top and bottom cues last.

[0050] 3. Calculate the covariance of the 4-channel post-prediction and remix-downmix as shown in equations [5] and [6].

number

[0051] In the WABC downmix example with 1 to 4 downmix channels, d and u represent the next channels, and the placeholder variables A, B, C can be any combination of the X, Y, Z channels in FoA. [Table 1]

[0052] 4. From these calculations, determine whether it is possible to cross-predict any remaining portion of the fully parametric channel from the transmitted residual channel. The additional C coefficients required are:

number

[0053] Thus, C has shape (1x2) for a 3-channel downmix and (2x1) for a 2-channel downmix. Some implementations of spatial noise filling do not require these C parameters and can set these parameters to 0. Other implementations of spatial noise filling may also include the C parameter.

[0054] 5. Calculate the remaining energy in the parameterized channels that need to be filled by the decorrelator. Upmix Channel Res uu The residual energy in is the actual energy R uu (Post-prediction) and Reproduced Cross-prediction Energy Reg uu is the difference between

number

[0055] Example Process The coefficient in P in equation

[10] determines to what extent the decorrelated components of W are used to recreate the A, B, and C channels before un-prediction and un-mixing.

[0056] 5 is a flow diagram of a process 500 for recreating background noise ambience in a multi-channel codec by generating spatial hole-filling noise, according to one embodiment. The process 500 can be implemented, for example, using the device architecture 600 described with reference to FIG.

[0057] The process 500 includes a step of calculating (501) a noise estimate based on a main downmix channel (e.g., W channel of FoA) generated from an input audio signal representing a spatial audio scene with a background noise ambience, a step of calculating (502) spectral shaping filter coefficients based on the noise estimate, a step of spectral shaping (503) a multi-channel noise signal using the spectral shaping filter coefficients and a noise distribution (e.g., Gaussian white noise) resulting in a diffuse multi-channel noise signal with uncorrelated channels (e.g., fully diffused), a step of spatially shaping (504) the diffuse uncorrelated multi-channel noise signal with uncorrelated channels based on the noise ambience of the spatial audio scene, and a step of applying (505) the spatially and spectrally shaped multi-channel noise signal to a multi-channel codec output to recreate the background noise ambience of the input spatial audio scene. Each of these steps has been described in detail with reference to Figures 1 to 4.

[0058] Exemplary System Architecture FIG. 6 shows a block diagram of an exemplary system 600 suitable for implementing the exemplary embodiment described with reference to FIGS. 1-5. The system 600 includes a central processing unit (CPU) 601 capable of executing various processes according to a program stored, for example, in a read-only memory (ROM) 602 or loaded, for example, from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, data required by the CPU 601 to execute the various processes is also stored as required. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0059] The following components are connected to the I / O interface 605: an input unit 606, which may include a keyboard, mouse, etc.; an output unit 607, which may include a display, such as a liquid crystal display (LCD), and one or more speakers; a storage unit 608, which may include a hard disk or other suitable storage device; and a communication unit 609, which may include a network interface card, such as a network card (e.g., wired or wireless).

[0060] In some embodiments, the input unit 606 includes one or more microphones in different locations (depending on the host device) that enable capture of audio signals in various formats (e.g., mono, stereo, spatial, immersive, or other suitable formats).

[0061] In some embodiments, the output unit 607 includes a system with a variable number of speakers. The output unit 607 can render audio signals in a variety of formats (e.g., mono, stereo, immersive, binaural, or other suitable formats).

[0062] The communication unit 609 is configured to communicate with other devices (e.g., via a network). If necessary, a drive 610 is also connected to the I / O interface 605. The drive 610 mounts a removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a flash drive, or other suitable removable medium, and if necessary, a computer program read therefrom is installed in the storage unit 608. Although the system 600 is described as including the above components, in a real application, it will be understood by those skilled in the art that some of these components can be added, removed, and / or replaced, and all such modifications or changes are all within the scope of the present disclosure.

[0063] Various aspects of the disclosed embodiments can be appreciated from the following enumerated example embodiments (EEE).

[0064] [EE1] 1. A method for recreating background noise ambience in a multi-channel codec by generating spatial hole-filling noise, the method comprising: calculating a noise estimate based on primary downmix channels generated from an input audio signal representing a spatial audio scene having a background noise ambience; 11. The method of claim 1, further comprising: calculating spectral shaping filter coefficients based on the noise estimates; spectrally shaping the multi-channel noise signal using the spectral shaping filter coefficients and a noise distribution, the spectral shaping resulting in a diffuse multi-channel noise signal having uncorrelated channels; spatially shaping the diffuse multi-channel noise signal having uncorrelated channels based on a noise ambience of the spatial audio scene; and applying the spatially and spectrally shaped multi-channel noise signal to a multi-channel codec output to synthesize the background noise ambience of the spatial audio scene.

[0065] [EE2] The method of claim EE1, wherein the spectral shaping is based on the spectral shape of the background noise ambience in a representation of a mid channel of a mid-side (M / S) signal or a W channel of a first order Ambisonics signal.

[0066] [EE3] 3. The method of claim EE1 or 2, wherein each channel of the uncorrelated channels has a similar spectral shape to the other channels.

[0067] [EE4] 4. The method of any one of claims EE1 to EE3, wherein spatially shaping the multi-channel noise signal is based on a covariance estimate of a decoded output of the multi-channel codec.

[0068] [EE5] 5. The method of any one of claims EE1 to EE4, wherein the spatial shaping of the multi-channel noise signal is based on spatial metadata extracted from the input audio signal.

[0069] [EE6] 6. The method of any one of claims 1 to 5, further comprising obtaining a spectral shape of the multi-channel noise signal by smoothing the gain of the multi-channel noise signal in time.

[0070] [EE7] 7. The method of any one of claims 1 to 6, wherein the dynamic range of the multi-channel noise signal is limited based on one or more adjustable thresholds.

[0071] [EE8] 8. The method according to any one of claims 1 to 7, wherein the multi-channel noise signal is added to a decoded multi-channel output to synthesize the input background noise ambience so as to mask spatial ambience collapse.

[0072] [EE9] 9. The method of any one of claims 1 to 8, wherein the multi-channel noise signal is added only to the parametrically upmixed multi-channel output.

[0073] [EE10] 10. The method of any one of claims EE1 to 9, wherein the multi-channel codec is an Immersive Voice and Audio Services (IVAS) codec.

[0074] [EE11] 11. The method of any one of claims EE1 to 10, wherein the spatial shaping and noise addition of the multi-channel noise signal is performed in the frequency banded domain or in the broadband domain.

[0075] [EE12] 12. The method of any one of claims EE1 to 11, wherein a multi-channel noise signal is added only to high frequencies.

[0076] [EE13] A system having one or more processors; and a non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform any one of the operations described in EE1 to EE12.

[0077] [EE14] A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the operations recited in any one of claims EE1 to 13.

[0078] According to example embodiments of the present disclosure, the above processes may be implemented as a computer software program or on a computer readable storage medium. For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine readable medium, the computer program including program code for performing the methods. In such an embodiment, the computer program may be downloaded and mounted from a network via a communication unit 609 and / or installed from a removable medium 611, as shown in FIG. 6.

[0079] In general, various exemplary embodiments of the present disclosure may be implemented in hardware or special purpose circuits (e.g., control circuits), software, logic, or any combination thereof. For example, the above-mentioned units may be executed by a control circuit (e.g., a CPU in combination with other components of FIG. 6), and thus the control circuit may perform the operations described in the present disclosure. Some aspects may be implemented in hardware, and other aspects may be implemented in firmware or software, which may be executed by a controller, microprocessor, or other computing device (e.g., control circuit). Although various aspects of the exemplary embodiments of the present disclosure are illustrated and described as block diagrams, flow charts, or using some other pictorial representations, it is understood that the blocks, apparatus, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof, as non-limiting examples.

[0080] Additionally, the various blocks illustrated in the flowcharts may be viewed as method steps and / or as resulting operations of computer program code and / or as multiple coupled logic circuit elements configured to perform the associated functions. For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program including program code configured to perform the methods described above.

[0081] In the context of this disclosure, a machine-readable medium may be any tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may be non-transitory and may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or any suitable combination thereof. More specific examples of machine-readable storage media include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0082] The computer program codes for carrying out the methods of the present disclosure may be written in any combination of one or more programming languages. These computer program codes may be provided to a processor of a general purpose computer, a special purpose computer, or other programmable data processing apparatus with control circuitry, such that when the computer program code is executed by the processor of the computer or other programmable data processing apparatus, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on the computer, partly on the computer, as a stand-alone software package, partly on the computer, partly on a remote computer, or entirely on a remote computer or server, or distributed across one or more remote computers and / or servers.

[0083] While this document contains many specific embodiment details, these should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as operating in certain combinations and may even initially be claimed as such, in some cases one or more features from a claimed combination may be deleted from the combination, and the claimed combination may be directed to a subcombination or variation of the subcombination. The logic flows depicted in the figures do not require the particular order depicted, or sequential order, to achieve desirable results. Furthermore, other steps may be provided or steps may be deleted from the described flows, and other components may be added to or deleted from the described systems. Accordingly, other embodiments are within the scope of the appended claims.

Claims

1. A method for generating a background noise ambience in a multi-channel codec, the method comprising: calculating, by at least one processor, a noise estimate of noise in a primary downmix channel from a primary downmix channel generated from an input audio signal representing a spatial audio scene with a background noise ambience; calculating, by the at least one processor, spectral shaping filter coefficients based on the noise estimate; spectrally shaping, by the at least one processor, a multi-channel noise signal using the spectral shaping filter coefficients and a noise distribution, the spectral shaping resulting in a diffuse multi-channel noise signal having uncorrelated channels; spatially shaping, by the at least one processor, the diffuse uncorrelated multi-channel noise signal having uncorrelated channels based on a noise ambience of the spatial audio scene; and outputting, by said at least one processor, spatially and spectrally shaped multi-channel noise to synthesize said background noise ambience of said spatial audio scene. method.

2. The method of claim 1 , wherein the spectral shaping is based on the spectral shape of the background noise ambience in a representation of the mid channel of a Mid-Side (M / S) signal or the W channel of a first order Ambisonics signal.

3. The method of claim 1 , wherein each of the uncorrelated channels of the multi-channel noise signal has the same spectral shape as the other channels.

4. The method of claim 1 , wherein spatially shaping the multi-channel noise signal is based on a covariance estimate of a decoded output of the multi-channel codec.

5. The method of claim 1 , wherein spatially shaping the multi-channel noise signal is based on spatial metadata extracted from the input audio signal.

6. The method of claim 1 , further comprising obtaining a spectral shape of the multi-channel noise signal by smoothing the gain of the multi-channel noise signal in time.

7. The method of claim 1 , wherein the dynamic range of the multi-channel noise signal is limited based on one or more adjustable thresholds.

8. The method of claim 1 , wherein the multi-channel noise signal is added to a decoded multi-channel output to synthesize the input background noise ambience so as to mask spatial ambience collapse.

9. The method of claim 1 , wherein the multi-channel noise signal is added only to a parametrically upmixed multi-channel output.

10. The method of claim 1 , wherein the multi-channel codec is an Immersive Voice and Audio Services (IVAS) codec.

11. The method of claim 1 , wherein the spatial shaping of the multi-channel noise signal is performed in the frequency band split domain or in the broadband domain.

12. The method of claim 1 , wherein the multi-channel noise signal is added only to high frequencies.

13. Calculating the noise estimate from the main downmix channel comprises: Read voice activity detector VAD decision flag; extracting a spectral shaping parameter if the VAD decision flag is zero. The method of claim 1.

14. one or more processors; a non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations; and wherein the operation comprises: calculating a noise estimate of noise in a primary downmix channel from the primary downmix channel generated from an input audio signal representing a spatial audio scene with a background noise ambience; calculating spectral shaping filter coefficients based on the noise estimate; spectrally shaping a multi-channel noise signal using the spectral shaping filter coefficients and a noise distribution, the spectral shaping resulting in a diffuse multi-channel noise signal having uncorrelated channels; spatially shaping the diffuse uncorrelated multi-channel noise signal having uncorrelated channels based on a noise ambience of the spatial audio scene; and outputting spatially and spectrally shaped multi-channel noise to synthesize the background noise ambience of the spatial audio scene. system.

15. A non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the operations recited in claim 1.

16. A method for generating a background noise ambience in a multi-channel codec, the method comprising: calculating, by at least one processor, a noise estimate of noise in a primary downmix channel from a primary downmix channel generated from an input audio signal representing a spatial audio scene with a background noise ambience; calculating, by the at least one processor, spectral shaping filter coefficients based on the noise estimate; spectrally shaping, by the at least one processor, a multi-channel noise signal using the spectral shaping filter coefficients and a noise distribution, the spectral shaping resulting in a diffuse multi-channel noise signal having uncorrelated channels; spatially shaping, by the at least one processor, the diffuse uncorrelated multi-channel noise signal having uncorrelated channels based on a noise ambience of the spatial audio scene; and outputting, by said at least one processor, spatially and spectrally shaped multi-channel noise to synthesize the background noise ambience of the spatial audio scene, regardless of whether a current frame of the input audio signal is active or inactive. method.

17. The method of claim 16, further comprising freezing spatial parameters for spatial shaping calculated in the last inactive frame during active frames.

18. The method of claim 16, wherein the multi-channel codec is an Immersive Voice and Audio Services (IVAS) codec.

19. One or more processors; and a non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the operations recited in claim 16.

20. A non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the operations recited in claim 16.