Parametric spatial audio rendering

Abstract

1. An apparatus comprising: means for obtaining a bitstream including encoded spatial metadata and an encoded transport audio signal; decoding the transport audio signal from the transport audio signal encoded in the bitstream; decoding spatial metadata from the spatial metadata encoded in the bitstream; generating an encoding metric; and generating the spatial audio signal from the transport audio signal based on the encoding metric and the spatial metadata.

Description

Technical Field

[0001] This application relates to apparatus and methods for spatial audio representation and rendering, and is not limited to audio representation for an audio decoder.

Background Art

[0002] Immersive audio decoders are implemented to support many operating points ranging from low bitrate operation to transparency. An example of such a decoder is the Immersive Voice and Audio Service (IVAS) decoder, which is designed for use in communication networks such as 3GPP 4G / 5G networks, including use in immersive services such as immersive voice and audio for virtual reality (VR). This audio decoder is expected to handle the encoding, decoding, and rendering of speech, music, and general audio. Further, it is expected to support channel-based audio inputs and scene-based audio inputs that include spatial information regarding sound fields and sound sources. The decoder is also expected to operate with low latency in order to enable conversational services and to support high error resilience under various transmission conditions.

[0003] Metadata-Assisted Spatial Audio (MASA) is one input format proposed for IVAS. It uses an audio signal together with corresponding spatial metadata. The spatial metadata defines the spatial aspect of the audio signal and can include parameters such as direction in a frequency band and direct-to-total energy ratio. A MASA stream can be obtained, for example, by capturing spatial audio with a microphone of a suitable capture device. For example, a mobile device including a number of microphones can be configured to capture microphone signals, and a set of spatial metadata can be estimated based on the captured microphone signals. A MASA stream can also be obtained from other sources such as certain spatial audio microphones (such as ambisonics), studio mixes (such as 5.1 audio channel mixes), or other content by appropriate format conversion. Summary of the Invention

[0004] According to a first aspect, there is provided an apparatus comprising means for obtaining a bitstream comprising an encoded spatial metadata and an encoded transport audio signal, decoding the transport audio signal from the bitstream encoded transport audio signal, decoding the spatial metadata from the encoded spatial metadata in the bitstream, generating an encoding metric, and generating a spatial audio signal from the transport audio signal based on the encoding metric and the spatial metadata.

[0005] This means may further include generating smoothing controls based on an encoding metric, and means for generating a spatial audio signal from a transport audio signal based on an encoding metric and spatial metadata may include generating a spatial audio signal from a transport audio signal based on smoothing controls and spatial metadata.

[0006] Means for generating a spatial audio signal from a transport audio signal based on an encoding metric and spatial metadata may further include modifying at least the energy ratio from the spatial metadata based on the encoding metric, and the spatial audio signal may be generated from the transport audio signal based on the modified energy ratio and spatial metadata.

[0007] Means for generating a spatial audio signal from a transport audio signal based on an encoding metric and spatial metadata may be for positioning directional sound in a direction determined by the spatial metadata, where the width of the directional sound is based on the encoding metric.

[0008] Means for generating a spatial audio signal from a transport audio signal based on an encoding metric and spatial metadata may include generating a covariance matrix from the transport audio signal and spatial metadata based on the encoding metric, generating a processing matrix based on the covariance matrix, and decorrelating and / or mixing the transport audio signals based on the processing matrix in order to generate a spatial audio signal.

[0009] The covariance matrix may include at least one of the following: an input covariance matrix representing the transport audio signal and a target covariance matrix representing the spatial audio signal.

[0010] Means for generating a covariance matrix from transport audio signals and spatial metadata may involve generating an input covariance matrix by measuring the transport audio signal in the time-frequency domain.

[0011] Means for generating a covariance matrix from transport audio signals and spatial metadata may be for generating a target covariance matrix based on spatial metadata and transport audio signal energy.

[0012] This method may further involve applying time averaging to the covariance matrix in order to generate an averaged covariance matrix, where time averaging is based on smoothing control, and generating a processing matrix based on the covariance matrix may involve generating a processing matrix from an averaged covariance matrix.

[0013] Means for generating a covariance matrix from transport audio signals and spatial metadata may be for generating a covariance matrix based on a modified energy ratio.

[0014] The means for generating a covariance matrix from a transport audio signal may be to generate the covariance matrix based on positioning directional sound in a direction determined by spatial metadata, where the width of the directional sound is based on the coding metric.

[0015] Means for generating a spatial audio signal from a transport audio signal based on an encoding metric and spatial metadata may include obtaining at least one direct-to-total energy ratio parameter based on spatial metadata; dividing the transport audio signal into a directional portion and an omnidirectional portion in a frequency band based on at least one direct-to-total energy ratio parameter from the spatial metadata; positioning the directional portion of the transport audio signal on at least one of a plurality of loudspeakers using amplitude panning; distributing and decorrelating the omnidirectional portion of the transport audio signal to all of the plurality of loudspeakers; and generating a combined audio signal based on the combination of the positioned directional portion of the transport audio signal and the omnidirectional portion of the transport audio signal.

[0016] The loudspeaker can be a virtual loudspeaker, and this means may further be used to generate a binaural spatial audio signal by applying a head-related transfer function to the combined audio signal.

[0017] A means for obtaining at least one direct-to-total energy ratio parameter based on spatial metadata may be one for obtaining at least one direct-to-total energy ratio from a modified energy ratio.

[0018] Means for positioning the directional portion of a transport audio signal on at least one of a plurality of loudspeakers using amplitude panning may be for positioning the directional portion of a transport audio signal on at least one of a plurality of loudspeakers using amplitude panning based on smoothing control.

[0019] A means for positioning directional sound in a direction determined by spatial metadata may involve using amplitude panning to position the directional sound on at least one of multiple loudspeakers, and the positioning width may be based on an encoding metric.

[0020] The means for generating coding metrics may be those for generating coding metrics based on the quality of representation of spatial metadata.

[0021] Means for generating encoded metrics may include encoded spatial metadata and means for generating encoded metrics from spatial metadata.

[0022] Means for generating encoded spatial metadata and an encoded metric from spatial metadata may include determining a first parameter indicating the number of bits intended or allocated to encode spatial parameters for a frame, determining a second parameter indicating the number of bits used after encoding spatial parameters has been performed for the frame, and generating an encoded metric as a ratio between the first parameter and the second parameter.

[0023] Spatial parameters can be directional indices representing quantized directional parameter values.

[0024] Means for generating an encoding metric may be for generating an encoding metric based on the quantization resolution of the spatial metadata and at least one of the ratios between at least two quantization resolutions of the spatial metadata.

[0025] A second aspect provides a method comprising: obtaining a bitstream containing encoded spatial metadata and encoded transport audio signals; decoding a transport audio signal from the transport audio signal encoded in the bitstream; decoding spatial metadata from the spatial metadata encoded in the bitstream; generating an encoded metric; and generating a spatial audio signal from the transport audio signal based on the encoded metric and spatial metadata.

[0026] This method may further include generating smoothing controls based on encoding metrics, and generating a spatial audio signal from a transport audio signal based on encoding metrics and spatial metadata may include generating a spatial audio signal from a transport audio signal based on smoothing controls and spatial metadata.

[0027] Generating a spatial audio signal from a transport audio signal based on an encoding metric and spatial metadata may involve modifying at least the energy ratio from the spatial metadata based on the encoding metric, and the spatial audio signal may be generated from the transport audio signal based on the modified energy ratio and spatial metadata.

[0028] Generating a spatial audio signal from a transport audio signal based on an encoding metric and spatial metadata may involve positioning a directional sound in a direction determined by the spatial metadata, with the width of the directional sound being based on the encoding metric.

[0029] Generating a spatial audio signal from a transport audio signal based on an encoding metric and spatial metadata may include generating a covariance matrix from the transport audio signal and spatial metadata based on the encoding metric, generating a processing matrix based on the covariance matrix, and decorrelating and / or mixing the transport audio signals based on the processing matrix to generate the spatial audio signal.

[0030] The covariance matrix may include at least one of the following: an input covariance matrix representing the transport audio signal and a target covariance matrix representing the spatial audio signal.

[0031] Generating a covariance matrix from transport audio signals and spatial metadata may include generating an input covariance matrix by measuring the transport audio signals in the time-frequency domain.

[0032] Generating a covariance matrix from transport audio signals and spatial metadata may include generating a target covariance matrix based on spatial metadata and transport audio signal energies.

[0033] This method may further include applying time averaging to the covariance matrix to generate an averaged covariance matrix, where time averaging is based on smoothing control, and generating a processing matrix based on the covariance matrix may include generating a processing matrix from the averaged covariance matrix.

[0034] Generating a covariance matrix from transport audio signals and spatial metadata may include generating a covariance matrix based on a modified energy ratio.

[0035] Generating a covariance matrix from a transport audio signal may involve generating a covariance matrix based on positioning directional sound in a direction determined by spatial metadata, where the width of the directional sound is based on the coding metric.

[0036] Generating a spatial audio signal from a transport audio signal based on an encoding metric and spatial metadata may include: obtaining at least one direct-to-total energy ratio parameter based on the spatial metadata; splitting the transport audio signal into a directional and an omnidirectional portion in the frequency band based on at least one direct-to-total energy ratio parameter from the spatial metadata; positioning the directional portion of the transport audio signal on at least one of a plurality of loudspeakers using amplitude panning; distributing and decorrelating the omnidirectional portion of the transport audio signal to all of the plurality of loudspeakers; and generating a combined audio signal based on the combination of the positioned directional portion of the transport audio signal and the omnidirectional portion of the transport audio signal.

[0037] The loudspeaker can be a virtual loudspeaker, and this method may further include generating a binaural spatial audio signal by applying a head-related transfer function to the combined audio signal.

[0038] Obtaining at least one direct-to-total energy ratio parameter based on spatial metadata may include obtaining at least one direct-to-total energy ratio from the modified energy ratio.

[0039] Using amplitude panning to position the directional portion of a transport audio signal on at least one of multiple loudspeakers may include using amplitude panning based on smoothing control to position the directional portion of a transport audio signal on at least one of multiple loudspeakers.

[0040] Positioning directional sound in a direction determined by spatial metadata may involve positioning the directional sound on at least one of multiple loudspeakers using amplitude panning, where the positioning width may be based on an encoding metric.

[0041] Generating coding metrics may include generating coding metrics based on the quality of representation of spatial metadata.

[0042] Generating encoded metrics can include encoding spatial metadata and generating encoded metrics from spatial metadata.

[0043] Generating encoded spatial metadata and encoding metrics from spatial metadata may include determining a first parameter indicating the number of bits intended or allocated to encode spatial parameters for a frame, determining a second parameter indicating the number of bits used after encoding spatial parameters has been performed for the frame, and generating encoding metrics as a ratio between the first and second parameters.

[0044] Spatial parameters can be directional indices representing quantized directional parameter values.

[0045] Generating an encoding metric may include generating an encoding metric based on the quantization resolution of the spatial metadata and at least one of the ratios between at least two quantization resolutions of the spatial metadata.

[0046] According to a third aspect, an apparatus is provided which includes at least one processor and at least one memory containing computer program code, wherein the at least one memory and the computer program code are configured to cause the apparatus to use the at least one processor to perform at least the following: acquire a bitstream containing encoded spatial metadata and encoded transport audio signals; decode transport audio signals from the transport audio signals encoded in the bitstream; decode spatial metadata from the spatial metadata encoded in the bitstream; generate an encoded metric; and generate a spatial audio signal from transport audio signals based on the encoded metric and spatial metadata.

[0047] The device may further generate smoothing controls based on an encoding metric, generate a spatial audio signal from a transport audio signal based on an encoding metric, and generate spatial metadata from a transport audio signal based on smoothing controls and spatial metadata.

[0048] A device capable of generating a spatial audio signal from a transport audio signal based on an encoding metric and spatial metadata may further modify at least the energy ratio from the spatial metadata based on the encoding metric, and the spatial audio signal may be generated from the transport audio signal based on the modified energy ratio and spatial metadata.

[0049] A device capable of generating a spatial audio signal from a transport audio signal based on an encoding metric and spatial metadata can position directional sound in a direction determined by the spatial metadata, with the width of the directional sound being based on the encoding metric.

[0050] A device capable of generating a spatial audio signal from a transport audio signal based on an encoding metric and spatial metadata may be capable of generating a covariance matrix from the transport audio signal and spatial metadata based on the encoding metric, generating a processing matrix based on the covariance matrix, and decorrelating and / or mixing the transport audio signal based on the processing matrix in order to generate a spatial audio signal.

[0051] The covariance matrix may include at least one of the following: an input covariance matrix representing the transport audio signal and a target covariance matrix representing the spatial audio signal.

[0052] A device capable of generating a covariance matrix from transport audio signals and spatial metadata can generate an input covariance matrix by measuring the transport audio signal in the time-frequency domain.

[0053] A device capable of generating a covariance matrix from transport audio signals and spatial metadata can generate a target covariance matrix based on the spatial metadata and transport audio signal energy.

[0054] This device can apply time averaging to the covariance matrix to generate an averaged covariance matrix, and the time averaging is based on smoothing control. The device can generate a processing matrix based on the covariance matrix, and can generate a processing matrix from the averaged covariance matrix.

[0055] A device capable of generating a covariance matrix from transport audio signals and spatial metadata may be capable of generating a covariance matrix based on a modified energy ratio.

[0056] A device capable of generating a covariance matrix from a transport audio signal may generate the covariance matrix based on positioning directional sound in a direction determined by spatial metadata, where the width of the directional sound is based on the coding metric.

[0057] A device capable of generating a spatial audio signal from a transport audio signal based on an encoding metric and spatial metadata may be capable of: obtaining at least one direct-to-total energy ratio parameter based on the spatial metadata; dividing the transport audio signal into a directional portion and an omnidirectional portion in a frequency band based on at least one direct-to-total energy ratio parameter from the spatial metadata; positioning the directional portion of the transport audio signal on at least one of a plurality of loudspeakers using amplitude panning; distributing and decorrelating the omnidirectional portion of the transport audio signal to all of the plurality of loudspeakers; and generating a combined audio signal based on the combination of the positioned directional portion of the transport audio signal and the omnidirectional portion of the transport audio signal.

[0058] The loudspeaker can be a virtual loudspeaker, and this device can further generate a binaural spatial audio signal by applying a head-related transfer function to the combined audio signal.

[0059] A device capable of obtaining at least one direct-to-total energy ratio parameter based on spatial metadata can obtain at least one direct-to-total energy ratio from the modified energy ratio.

[0060] A device that uses amplitude panning to position the directional portion of a transport audio signal on at least one of a plurality of loudspeakers may be able to position the directional portion of a transport audio signal on at least one of a plurality of loudspeakers using amplitude panning based on smoothing control.

[0061] A device capable of positioning directional sound in a direction determined by spatial metadata can use amplitude panning to position the directional sound on at least one of several loudspeakers, and the positioning width can be based on an encoding metric.

[0062] A device capable of generating coding metrics may be able to generate coding metrics based on the quality of the spatial metadata representation.

[0063] A device capable of generating encoded metrics can generate encoded metrics from encoded spatial metadata and from spatial metadata.

[0064] A device capable of generating encoded spatial metadata and encoding metrics from spatial metadata may be configured to determine a first parameter indicating the number of bits intended or allocated to encode spatial parameters for a frame, a second parameter indicating the number of bits used after encoding spatial parameters has been performed for the frame, and to generate encoding metrics as a ratio between the first parameter and the second parameter.

[0065] Spatial parameters can be directional indices representing quantized directional parameter values.

[0066] A device capable of generating an encoding metric may generate the encoding metric based on the quantization resolution of the spatial metadata and at least one of the ratios between at least two quantization resolutions of the spatial metadata.

[0067] According to a fourth aspect, an apparatus is provided that includes: an acquisition circuit configured to acquire a bitstream containing encoded spatial metadata and an encoded transport audio signal; a decoding circuit configured to decode a transport audio signal from the transport audio signal encoded in the bitstream; a decoding circuit configured to decode spatial metadata from the spatial metadata encoded in the bitstream; a generation circuit configured to generate an encoded metric; and a generation circuit configured to generate a spatial audio signal from a transport audio signal based on the encoded metric and the spatial metadata.

[0068] According to a fifth aspect, a computer program [or a computer-readable medium containing program instructions] is provided, which includes instructions for causing a device to perform at least the following: acquire a bitstream containing encoded spatial metadata and encoded transport audio signals; decode transport audio signals from the transport audio signals encoded in the bitstream; decode spatial metadata from the spatial metadata encoded in the bitstream; generate an encoded metric; and generate spatial audio signals from transport audio signals based on the encoded metric and spatial metadata.

[0069] According to the sixth aspect, a non-temporary computer-readable medium is provided which includes program instructions for causing the device to perform at least the following: acquire a bitstream containing encoded spatial metadata and encoded transport audio signals; decode transport audio signals from the transport audio signals encoded in the bitstream; decode spatial metadata from the spatial metadata encoded in the bitstream; generate an encoded metric; and generate spatial audio signals from transport audio signals based on the encoded metric and spatial metadata.

[0070] According to a seventh aspect, an apparatus is provided that includes means for acquiring a bitstream containing encoded spatial metadata and encoded transport audio signals; means for decoding transport audio signals from the transport audio signals encoded in the bitstream; means for decoding spatial metadata from the spatial metadata encoded in the bitstream; means for generating an encoding metric; and means for generating a spatial audio signal from transport audio signals based on the encoding metric and spatial metadata.

[0071] According to the eighth aspect, a computer-readable medium is provided which includes program instructions for causing the device to perform at least the following: acquire a bitstream containing encoded spatial metadata and encoded transport audio signals; decode transport audio signals from the transport audio signals encoded in the bitstream; decode spatial metadata from the spatial metadata encoded in the bitstream; generate an encoded metric; and generate spatial audio signals from transport audio signals based on the encoded metric and spatial metadata.

[0072] An apparatus including means for performing the operation of the method described above.

[0073] A device configured to perform the operation described above.

[0074] A computer program that contains program instructions for causing a computer to perform the methods described above.

[0075] A computer program product stored on a medium can cause the device to perform the actions described herein.

[0076] The electronic device may include devices such as those described herein.

[0077] The chipset may include devices such as those described herein.

[0078] The embodiments of this application aim to address problems related to the latest technology.

[0079] For a better understanding of this application, the attached drawings are now referred to as examples. [Brief explanation of the drawing]

[0080] [Figure 1] This diagram schematically illustrates a system of an apparatus suitable for carrying out several embodiments. [Figure 2] This figure schematically shows a decoder as shown in the system of the device shown in Figure 1, according to several embodiments. [Figure 3] Figure 2 shows a flowchart illustrating the operation of an exemplary decoder according to several embodiments. [Figure 4] This figure schematically illustrates an exemplary synthesis processor, as shown in Figure 2, according to several embodiments. [Figure 5] Figure 4 shows a flowchart illustrating the operation of an exemplary synthesis processor according to several embodiments. [Figure 6] This figure shows an example device suitable for realizing the apparatus shown in the previous figure. [Modes for carrying out the invention]

[0081] The following describes in further detail suitable devices and possible mechanisms for decoding a parametric spatial audio stream containing transport audio signals and spatial metadata.

[0082] As discussed above, Metadata-Assisted Spatial Audio (MASA) is an example of a suitable parametric spatial audio format and representation for use as an input format for IVAS.

[0083] MASA can define an audio scene as, for example, an audio representation consisting of "N channels + spatial metadata." It is a scene-based audio format particularly well-suited for spatial audio capture on practical devices such as smartphones. This concept is for describing acoustic scenes in terms of time and frequency-varying sound source directions and, for example, energy ratios. Sound energy not defined (described) by direction is described as diffuse (coming from all directions).

[0084] As discussed above, spatial metadata associated with an audio signal can include numerous parameters for each time-frequency tile (multiple directions, and for each direction, direct-to-total ratio, diffuse coherence, distance, etc.). Spatial metadata can further include other parameters or be associated with other parameters that can be considered omnidirectional (surround coherence, diffuse-to-total energy ratio, residual-total energy ratio, etc.), but when combined with directional parameters, it can be used to define the characteristics of the audio scene. For example, a reasonable design choice that can produce a good quality output is one in which the spatial metadata includes one or more directions determined for each time-frequency subframe (and for each direction, direct-to-total ratio, diffuse coherence, distance values, etc.).

[0085] As described above, the parametric spatial metadata representation can use a number of simultaneous spatial directions. In MASA, the maximum number of proposed simultaneous directions is 2. For each simultaneous direction, there may be associated parameters such as the direction index, direct-to-total ratio, diffusion coherence, and distance. In some embodiments, other parameters such as the diffusion-to-total energy ratio, surround coherence, and residual-total energy ratio are defined.

[0086] At very low bitrates (e.g., around 13.2-16.4 kbps), there are very few bits available to encode metadata. For example, to obtain a sufficient bitrate for an audio signal codec, only about 3 kbps can be used for encoding metadata.

[0087] In many cases, only a few bits can be used per value (e.g., direction parameter) to achieve sufficient frequency and time resolution (e.g., having five frequency bands and a time resolution of 20 milliseconds). In practice, this means that the quantization steps are relatively large. Thus, for example, in a given time-frequency tile, the quantization points are at azimuth angles of 0, ±45, ±90, ±135, and 180 degrees.

[0088] The directional resolution of human hearing is approximately 1-2 degrees in the azimuth direction. Therefore, a jump of, for example, 0-45 degrees can be easily perceived, but it degrades the perceived audio quality and makes playback unnatural.

[0089] This can be mitigated by adding temporal smoothing to the rendering. For example, when rendering binaural signals from microphone signals, the processed values ​​(based on spatial metadata) can be temporally averaged using a first-order IIR (Infinite Impulse Response) filter. Depending on the rendering scheme, in other configurations, temporal averaging is performed on the measured multi-microphone input covariance matrix (or, in the case of a spatial audio decoder, the measured covariance matrix of the transport audio signal, or the measured covariance matrix of any other type of input signal) and the metadata target covariance matrix, and the processed values ​​are then determined based on these averaging matrices.

[0090] As a result, jumps caused by low directional resolution can be mitigated, and instead, the direction is perceived as changing smoothly. However, adding smoothing also adds sluggishness and delay to rendering. For example, sudden sound source activity in a particular direction may only be reproduced vaguely in that direction, at least for the initial moments of the activity. The more smoothing is used, the slower the rendering becomes, and the greater the overall delay in the experience. Therefore, the amount of temporal smoothing applied is a compromise between fast and responsive rendering and the reduction of artifacts from rapidly changing directions.

[0091] Even in the worst-case scenario (for example, when encoding a complex acoustic scene that is difficult to encode efficiently at a low bitrate), if smoothing is applied (adjusted) to handle artifacts from the jump direction, the amount of time smoothing required can be quite large, resulting in renderings that are quite slow and plagued by latency. When this type of rendering is applied even at higher bitrates where directional resolution is significantly better, suboptimal quality is experienced.

[0092] On the other hand, if temporal smoothing is optimized for higher bitrates and adjusted to have fast, responsive rendering, playback may be perceived as spatially unstable at lower bitrates.

[0093] Furthermore, the directional resolution achieved is generally not fixed at a given bitrate, but depends on how well the encoding can compress the content data. Therefore, techniques with a fixed smoothing time constant will always produce results that fall short of optimal, as the directional resolution can change with time and frequency.

[0094] Time smoothing of processing parameters is also used in various parametric space audio processing methods. For example, in Pulkki, V. (2007). Spatial sound reproduction with directional audio coding. Journal of the Audio Engineering Society, 55(6), 503-516, directional audio coding (DirAC) uses time smoothing applied to loudspeaker panning gain. Vilkamo, J., Backstrom, T., & Kuntz, A. (2013). Optimized covariance domain framework for time-frequency processing of spatial audio. Journal of the Audio Engineering Society, 61(6), 403-411 proposes a method for determining spatial audio processing using a covariance matrix. For example, Vilkamo, J., & Pulkki, V. (2013). Minimization of decorrelator artifacts in directional audio coding by covariance domain rendering. Journal of the Audio Engineering Society, 61(9), 637-646 describes the execution of DirAC rendering, in which the covariance matrix is ​​averaged by a first-order IIR filter so that the covariance matrix is ​​more averaged at lower frequencies. Averaging the covariance matrix results in a smoother overall processing gain over time, avoiding overly abrupt processing, i.e., artifacts. However, these methods do not consider the loss of parameter precision in coding, nor do they vary the amount of averaging over time.

[0095] Accordingly, the concepts discussed in more detail in the following embodiments describe apparatuses and methods that enable optimized spatial audio rendering based on the encoding quality of spatial metadata, with respect to the rendering of encoded parametric spatial audio (i.e., audio signals and spatial metadata). This can be achieved, in some embodiments, by taking an (encoded) spatial audio stream (including audio signals and spatial metadata), determining a metric that describes the achieved encoding quality of the metadata for a particular time and / or frequency interval, adjusting the renderer based on that metric (e.g., adjusting the amount of temporal smoothing and / or environmental processing in a binaural renderer), and rendering the spatial audio using the adjusted renderer based on the audio signals and spatial metadata.

[0096] The embodiments will be described with respect to the exemplary capture (or encoder / analyzer) and playback (or decoder / synthesizer) device or system 100 shown in Figure 1. In the following examples, the audio signal input is from a microphone array; however, the audio input can be any suitable audio input format, and it will be understood that this specification will detail below any differences in processing that occur when different input formats are used.

[0097] System 100 is represented by the capture section and the playback (decoder / synthesizer) section.

[0098] In some embodiments, the capture section includes a microphone array audio signal input 102. The input audio signal can be from any suitable source, e.g., two or more microphones attached to a mobile phone, other microphone arrays, e.g., B-format microphones, Eigenmike. In some embodiments, as described above, the input can be any suitable audio signal input such as ambisonic signals, e.g., first-order ambisonics (FOA), higher-order ambisonics (HOA), or loudspeaker surround mix and / or object.

[0099] The microphone array audio signal input 102 may be provided to the microphone array front end 103. In some embodiments, the microphone array front end is configured to perform an analysis processor function configured to generate or determine appropriate (spatial) metadata related to the audio signal, and to perform an appropriate transport signal generator function to generate a transport audio signal.

[0100] Therefore, the analysis processor function is configured to perform spatial analysis on the input audio signal, yielding appropriate spatial metadata 106 in the frequency band. For all of the aforementioned input types, there are known methods for generating appropriate spatial metadata, e.g., direction and direct-to-total energy ratio (or diffusion, i.e., similar parameters such as ambient-to-total ratio) in the frequency band. These methods are not detailed herein, however, some examples may include performing an appropriate time-frequency conversion on the input signal, then, in the case where the input is a mobile phone microphone array, estimating the delay value between microphone pairs that maximizes the correlation between microphone pairs in the frequency band, formulating a direction value corresponding to that delay (as described in UK Patent Application No. 1619573.7 and PCT Patent Application No. PCT / FI2017 / 050778), and formulating a ratio parameter based on the correlation value. The direct-to-total energy ratio parameter of a multi-channel captured microphone array signal can be estimated based on a normalized cross-correlation parameter cor'(k,n) between microphone pairs in band k, where the value of the cross-correlation parameter is between -1 and 1. The direct-to-total energy ratio parameter r(k,n) is,

[0101]

number

[0102] Metadata can take various forms and, in some embodiments, may include spatial metadata and other metadata. Typical parameterization for spatial metadata is one directional parameter in each frequency band characterized as an azimuth value φ(k,n) and an elevation value θ(k,n), and an associated direct-to-total energy ratio r(k,n) in each frequency band, where k is the frequency band index and n is the time frame index.

[0103] In some embodiments, the parameters generated may differ for each frequency band. For example, in band X, all parameters may be generated and transmitted, while in band Y, only one of the parameters may be generated and transmitted, and furthermore, in band Z, no parameters may be generated or transmitted at all. A practical example of this might be that in some frequency bands, such as the highest band, some parameters are not needed for perceptual reasons.

[0104] In some embodiments, when the audio input is an FOA signal or a B-format microphone, the analysis processor function can be configured to determine parameters such as the intensity vector from which the directional parameter is obtained, and to determine the ratio parameter by comparing the intensity vector length with the overall sound field energy estimate. This method is known in the literature as directional audio coding (DirAC).

[0105] In some embodiments, if the input is an HOA signal, the analysis processor function can take an FOA subset of the signal and use the method described above, or it can divide the HOA signal into a number of sectors and use the method described above in each of the number of sectors. This sector-based method is known in the literature as higher-order DirAC (HO-DirAC). In this case, there are two or more simultaneous directional parameters for each frequency band.

[0106] In some embodiments, when the input format is a loudspeaker surround mix and / or object, the analysis processor function can be configured to convert the signal to an FOA signal (via the use of spherical harmonic coding gain) and analyze the direction and ratio parameters as described above.

[0107] Therefore, the output of the analysis processor function is (spatial) metadata 106 determined in the frequency band. The (spatial) metadata 106 may include direction and energy ratio in the frequency band, but may also have any of the metadata types listed above. The (spatial) metadata 106 may change over time and across frequency.

[0108] In some embodiments, the analysis function is performed outside of system 100. For example, in some embodiments, spatial metadata related to the input audio signal may be provided to encoder 107 as a separate bitstream. In some embodiments, spatial metadata may be provided as a set of spatial (directional) index values.

[0109] The microphone array front end 103 is further configured to perform a transport signal generator function to generate an appropriate transport audio signal 104, as described above. The transport signal generator function is configured to receive an input audio signal, which may be, for example, a microphone array audio signal 102, and generate a transport audio signal 104. The transport audio signal may be a multi-channel, stereo, binaural, or mono audio signal. The generation of the transport audio signal 104 may be performed using any suitable method, such as those summarized below.

[0110] When the input is a microphone array audio signal, the transport signal generator function can select the left and right microphone pairs and apply appropriate processing such as automatic gain control, microphone noise cancellation, wind noise cancellation, and equalization to the signal pairs.

[0111] If the input is an FOA / HOA signal or a B-format microphone, the transport signal 104 may be a directional beam signal directed left-right, such as two opposing cardioid signals.

[0112] If the input is a loudspeaker surround mix and / or object, the transport signal 104 can be a downmix signal that combines the left channel into a left downmix channel, and the same for the right channel, and adds the center channel to both transport channels with appropriate gain.

[0113] In some embodiments, the transport signal 104 is an input audio signal, such as a microphone array audio signal. The number of transport channels can also be any suitable number (rather than one or two channels as discussed in the example).

[0114] In some embodiments, the capture portion may include an encoder 107. The encoder 107 may be configured to receive a transport audio signal 104 and spatial metadata 106. The encoder 107 may further be configured to generate a bitstream 108 containing metadata information and an encoded or compressed format of the transport audio signal.

[0115] The encoder 107 can be implemented, for example, as an IVAS encoder or any other suitable encoder. In such embodiments, the encoder 107 is configured to encode an audio signal and metadata to form an IVAS bitstream.

[0116] Next, this bitstream 108 can be transmitted / stored, as shown by the dashed line.

[0117] System 100 may further include a decoder 109. The decoder 109 is configured to receive, extract, or otherwise acquire a bitstream 108 and generate an appropriate spatial audio signal 110 from the bitstream to be presented to a listener / listener playback device.

[0118] Therefore, the decoder 109 is configured to receive the bitstream 108, multiplex the encoded stream, and then decode the audio signal to obtain the transport signal and metadata.

[0119] The decoder 109 can further be configured to produce a spatial audio signal output 110, for example, a binaural audio signal that can be played back by headphones, from the transport audio signal and spatial metadata.

[0120] Referring to Figure 1, it has been stated that part of the function of encoder 107 is to encode spatial audio parameters (MASA), in other words, spatial metadata 106. For example, directional values ​​(azimuth angle φ(k,n) and elevation angle value θ(k,n)) can first be quantized according to a spherical quantization scheme. Such a scheme can be found in Japanese Patent Publication EP3707706. In general, each type of spatial audio parameter is first quantized in order to obtain a quantization index.

[0121] The resulting quantization index from spatial audio parameters (e.g., MASA parameters) can then be entropy-encoded at different encoding rates, depending on a coefficient that defines the number of bits allocated to the task. As described above, a codec system can use several different encoding rates, and this also applies to the encoding of the spatial audio parameter index.

[0122] For example, a general framework for encoding the index values ​​of the direction parameter of all TF tiles within a frame can take the following form: Input: Index of quantized directional parameters (azimuth and elevation) and the number of bits allowed B allowed 1. Use EC1 to encode parameters 2. bit_EC1 allowed In that case a. Encode using EC1 3. Otherwise a. Use bandwise encoded EC2 (with potential quantization resolution reduction). b. Bit_EC2 allowed In that case i. Encode using EC2 c. Otherwise i. Reduce quantization resolution ii. Use EC3 d. End if 4. End if

[0123] ​​In the above, EC1 corresponds to a first entropy coding scheme that can encode the azimuth and elevation indices separately. This scheme uses an optimized fixed-value average index, which is subtracted from each index, thereby yielding a difference index for each directional index. In these schemes, there is one average for azimuth and one average for elevation. For those frequency tiles with an energy ratio greater than a threshold, the average is taken over the entire frame (for those frequency tiles, it is more efficient to send the combined elevation and azimuth indices together for one time frequency tile). Each resulting difference index can then be converted to a positive value and then entropy coded using the Golomb-Rice scheme. The optimized average index can then be entropy coded further for transmission to the decoder.

[0124] EC2 supports a second entropy coding scheme that encodes the differential index with lower resolution than EC1. Details of a suitable second entropy coding scheme can be found in patent publication WO2021 / 048468.

[0125] EC3 corresponds to a third entropy coding scheme that encodes differential indices with lower resolution than EC2. In this respect, EC3 may constitute the lowest resolution quantization scheme in the general framework described above. Details of suitable schemes can be found in Japanese Patent Publication EP3861548.

[0126] From the general framework described above, the selection of the coding rate (and therefore the coding scheme) is determined by parameter B, which indicates the number of bits allowed for coding the direction index of the frame. allowed It can be seen that this can be partially determined by B. allowed This can be a parameter for determining the encoding system based on the overall operating point / bitrate of the encoder in a particular time frame.

[0127] As can be seen from the above, parameter Ballowed Using allowed , the entropy encoding method can be determined by essentially checking whether the bits required for the entropy encoding method are less than parameter B allowed This check process is executed in order from the larger of the bits required for the entropy encoding method. The result of the check process is that the highest-order entropy encoding method (of the encoded bits) that satisfies the constraint of B allowed is selected.

[0128] For example, if the number of bits (bits_EC1) required for the first entropy encoding method EC1 is less than B allowed the first entropy encoding method is used. However, if it is determined that the bits required for EC1 are greater than the constraint B allowed then the number of bits (bits_EC2) required for the second entropy encoding method EC2 is compared with B allowed In the second check, if it is shown that the bits required for EC2 are less than B allowed the second entropy encoding method EC2 is used to entropy-encode the direction index of the frame. However, if the second check shows that the bits required for EC2 are greater than (or equal to) B allowed then the third entropy encoding method EC3 is selected to encode the direction index.

[0129] The general framework described above can be extended to any number of coding rates, and each entropy encoding method is selected according to the number of bits required (bits_ECn) and the allowable bits B allowed

[0130] FIG. 2 shows decoder 109 in more detail.

[0131] ​In some embodiments, the decoder 109 includes a demultiplexer 201 configured to accept and multiplex the bitstream 108 in order to obtain an encoded transport audio signal 204 and encoded spatial audio parameter metadata (MASA metadata) 202.

[0132] In some embodiments, the decoder 109 further includes a transport audio signal decoder 205 configured to decode the encoded transport audio signal 204, thereby producing a decoded transport audio signal stream 210 that is passed to a spatial synthesizer 207. The decoding process performed by the transport audio signal decoder 205 can be any suitable audio signal decoding scheme for the encoded transport audio signal, such as an EVS decoder when EVS encoding is used.

[0133] Figure 2 further shows a metadata decoder 203 configured to accept encoded spatial metadata (encoded spatial audio parameters) 202, decode the metadata to produce decoded spatial metadata 206. Decoding of the spatial metadata 206 is performed as the inverse of metadata encoding, and therefore, in some embodiments, as an entropy decoding operation based on the determination of the entropy encoding mode performed. In some embodiments, as part of the decoding process, the metadata decoder 203 is configured to further produce a quality of encoding metric 208 of the decoded spatial audio parameters / metadata. The quality of encoding metric 208 is configured to be passed together with the decoded spatial metadata 206 to the spatial synthesizer 207.

[0134] The spatial metadata (audio parameters) encoding quality metric 208 can be used as an indicator of entropy coding performance, describing how well the spatial metadata was encoded.

[0135] The metric can be obtained as frequency-dependent Ξ(k,n) (where k is the frequency band index and n is the time frame (or subframe) index) or frequency-independent Ξ(n).

[0136] The following example describes a frequency-independent variant Ξ(n), but a frequency-dependent embodiment is implemented in a similar manner. The metric scale can be any suitable range. The following example describes a metric range of 0 to 1, where 1 means that the encoding achieved the target resolution of the metadata, and 0 means that, in effect, no information could be transmitted. Values ​​between 0 and 1 mean that the target resolution was not achieved, but at least some information could be transmitted. In practice, Ξ(n) is generally between 0.5 and 1.

[0137] The frequency-independent coding quality metric Ξ(n)208 can be defined, for example, as the relationship between the target bit budget and the actual bits used in spatially oriented coding. In some embodiments,

number

[0138] In some further embodiments, B used (n) The value can be obtained as the actual bit used. In such an embodiment, in the decoder, B used The (n) value is estimated after entropy coding-based bit transmission from subband to subband. Therefore, in such embodiments, it is not necessary to explicitly indicate the number of bits used in the bitstream. In some coding methods, B used (n) is always B target (n) is equal to Ξ(n), and the resulting Ξ(n) is equal to 1. When these methods are used, Ξ(n) can be directly assigned the value 1. However, these methods have a total allowable bit budget B allowed (n) is the target bitrate B targetIt cannot be used if it is less than (n). In such embodiments, the reduction of quantization precision may be carried out in various ways, and the method described in GB1811071.8 is a suitable example. As a side note, since there are some readjustments of bit assignments per subband or time-frequency tile during the encoding itself, the resulting bit budget should be calculated as the sum of the individual bit assignments of the subband / time-frequency-wavenumber tile after the decoding procedure. These reduction methods are used for the actual bit rate B used (n) can be reported, and that is the target bitrate B target If it is less than (n), then the metric Ξ(n) will also be less than 1. Therefore, the metric in this case is actually the allowed bit budget B. allowed To fit (n), the target bit budget B is reduced by reducing the precision of spatial quantization. target This will tell us how many bits had to be reduced from (n). used (n) is B allowed It should be noted that it is acceptable for it to be close to (n), and it does not necessarily have to be equal to it. In some exemplary embodiments, B used (n) If certain conditions occur, B allowed (n) can be greater than this. Nevertheless, B used (n)≦B target (n) should still be generally applicable, and the above formula is used. Furthermore, in some embodiments, B used (n) is B allowed If (n) is greater than 208, the coding quality metric Ξ(n)208 is set to 1.

[0139] In some embodiments, the decoder includes a spatial combiner 207 configured to receive an encoded quality metric Ξ(n) 208, decoded spatial metadata 206, and a decoded transport audio signal 210. The spatial combiner 207 is then configured to render a spatial audio output signal, such as a binaural audio signal. In such embodiments, the spatial combiner 207 is configured to use the encoded quality metric 208 to adjust the rendering, thereby enabling optimized spatial audio quality with both good quality spatial metadata (e.g., commonly found at high bitrates) and compromised spatial metadata (e.g., sometimes with important signal content at low bitrates).

[0140] Figure 3 shows a flowchart illustrating an exemplary operation performed by a decoder as shown in Figure 2, according to several embodiments.

[0141] For example, Figure 3 shows that the bitstream is received in step 301.

[0142] Next, the bitstream is multiplexed and separated by step 303 into encoded spatial metadata and encoded transport audio signals, as shown in Figure 3.

[0143] Next, the encoded transport audio signal is decoded by step 306 to generate a transport audio signal as shown in Figure 3.

[0144] The encoded spatial metadata is further decoded by step 305 to generate spatial metadata as shown in Figure 3.

[0145] In addition, an encoding quality metric is generated by step 307, as shown in Figure 3.

[0146] Next, the output spatial audio signal is spatially synthesized from the transport audio signal based on spatial metadata and encoding quality metrics, as shown in Figure 3, by step 309.

[0147] Furthermore, the output spatial audio signal is then output by step 311 as shown in Figure 3.

[0148] With respect to Figure 4, several embodiments of the spatial synthesizer 207 are shown in more detail.

[0149] In some embodiments, the spatial combiner 207 includes a forward filter bank (time-frequency converter) 401. The forward filter bank (time-frequency converter) 401 is configured to receive the (time-domain) decoded transport audio signal 210 and convert it to the time-frequency domain. Suitable forward filters or transforms include, for example, the Short-Time Fourier Transform (STFT) and the Complex Modulation Quadrature Mirror Filter Bank (QMF). The resulting signal is x i It can be expressed as (b,n), where i is the channel index, b is the frequency bin index of the time-frequency transform, and n is the time index. The time-frequency signal is represented in vector form, for example, here (for example, for a 2-channel signal, the vector form is as follows):

[0150]

number

[0151] In some embodiments, the time-frequency transport signal 402 may be provided to an input and target (or output) covariance matrix decisioner 403, a processing matrix decisioner 407, and a decorrelator and mixer 409.

[0152] In some embodiments, the spatial synthesizer 207 includes an input and target (or output) covariance matrix determinator 403. The input and target covariance matrix determinator 403 is configured to receive decoded spatial metadata 206 and a time-frequency transport signal 402 and determine a covariance matrix. The determined covariance matrix includes an input covariance matrix representing the time-frequency transport signal 402 and an output covariance matrix representing the time-frequency spatial audio signal 410. The input covariance matrix is ​​measured from the time-frequency transport signal 402 and may be represented as a column vector x(b,t), where b is the frequency bin index, t is the time-frequency signal time index, and the rows represent the transport signal channels. In some embodiments,

[0153]

number

[0154] In some embodiments, the target covariance matrix is ​​determined based on spatial metadata and total signal energy. The total signal energy E(b,n) is C x It can be obtained as the average of the diagonal values ​​of (b,n). Then, in some embodiments, the spatial metadata includes the direction DOA(k,n) and the direct-to-total ratio parameter r(k,n). Note that the bandwidth index k is the bandwidth index where bin b exists. If the spatial audio output is a binaural signal, the target covariance matrix is: C y (b,n)=E(b,n)r(k,n)h(b,DOA(k,n))h H (b,DOA(k,n))+E(b,n)(1-r(k,n))C d (b) It can be determined as follows, where h(b,DOA(k,n)) is a column vector of length 2 with complex values, which is the head-related transfer function column vector for bins b and DOA(k,n), and corresponds to the HRTF amplitude and phase of the left and right ears. At high frequencies, the phase difference is not required for the reasons of perception at high frequencies, so the HRTF value can also be a real number. The HRTF for a given direction and frequency can be determined based on any suitable method. d (b) is the diffusion-field binaural covariance matrix, which can be determined, for example, by taking a spatially uniform set of HRTFs in the offline phase, independently formulating their covariance matrices, and averaging the results.

[0155] Next, the input covariance matrix C x (b,n) and the target covariance matrix C y (b,n) can be output to the time averager 405 as the covariance matrix 404.

[0156] The example above illustrates the use of direction and ratio. The procedure for generating the target covariance matrix is ​​described more broadly in GB2572650, which also describes spatial coherence parameters in addition to direction and ratio, and further covers output types other than binaural output.

[0157] In some embodiments, the spatial synthesizer 207 includes a time averager 405. The time averager 405 is a covariance matrix 404 C x (b,n) and C y The time averager 405 is configured to accept (b,n) and the coding quality metric 208 Ξ(n). The averaged covariance matrix C' is defined as a matrix of zero when n<0. x (b,n) and C' y It consists of memory (b,n). In some embodiments, the time averager 405 operates according to infinite impulse response (IIR) type averaging and is configured to utilize energy values ​​in the regression process.

[0158] For example, E' x (b,n) to C' x Expressed as the average of the diagonals of (b,n), E x (b,n) C x Expressed as the average of the diagonals of (b,n), and equivalently for the subscript y. The averaging operation is performed in some embodiments as follows:

number

[0159] In some embodiments, operators α(Ξ(n)) and β(Ξ(n)) control the forget / remember rate of the IIR processing based on the quality of the encoded value Ξ(n). Value α(Ξ(n)) controls how much of the previous signal energy is retained in memory (i.e., how much is averaged), and β(Ξ(n)) controls how much the new frame affects the averaged covariance matrix. In some embodiments, encoding quality and quality variation over time are used to utilize these control parameters. In some embodiments, it is assumed that encoding provides irregularly good or bad encoding due to different bit budget allocations, and therefore these parameters are used. α(Ξ(n))=c+(1-Ξ(n) p ) β(Ξ(n))=Ξ(n) p It is defined as follows: c and p are coefficients that control the amount of smoothing (e.g., c=2 and p=2).

[0160] In some embodiments, only α or β is modified based on the coding quality metric Ξ(n), while a fixed value is used for the other. Furthermore, other methods based on Ξ(n) can also be used to perform smoothing.

[0161] Next, the time averager 405 is the averaged covariance matrix 406 C' x (b,n) and C' y It can be configured to output (b,n).

[0162] In some embodiments, the spatial synthesizer 207 includes a processing matrix decisioner 407. The processing matrix decisioner 407 uses the averaged covariance matrix C' x (b,n) and C' y (b,n)406 and time-frequency transport signal 402 are received, and the processing matrix M(b,n) and M r The (b,n)408 is configured to determine the processing matrix. The determination of the processing matrix based on the covariance matrix can be carried out based on appropriate methods, such as those described in Juha Vilkamo, Tom Backstrom, and Achim Kuntz. “Optimized covariance domain framework for time-frequency processing of spatial audio.” Journal of the Audio Engineering Society 61.6 (2013): 403-411. In this disclosure, the measured covariance matrix C' x The determination of the mixing matrix for processing the audio signal using (b,n) is determined by the determined target covariance matrix C'. y The process is carried out to reach (b,n). This method has been used in various use cases in the literature, including the generation of binaural or surround loudspeaker signals. When formulating the processing matrix, this method further determines a prototype matrix, which is a matrix that tells the optimization procedure what kind of signal is generally intended for each output (with the constraint that the output must reach the target covariance matrix). For example, in some embodiments that implement binaural sound reproduction, the prototype matrix is:

[0163]

number

[0164]

number

[0165] In some embodiments, the spatial combiner 207 processes the time-frequency transport signal x(b,t)402 and the processing matrices M(b,n) and M r The system includes a decorrelator and mixer 409 configured to receive (b,n)408. The decorrelator and mixer 409 first processes the input time-frequency transport signal using the prototype matrix determined in the processing matrix decisioner 407, then decorrelates the result, thereby obtaining the decorrelated signal x D The uncorrelator and mixer 409 are then configured to generate (b,t). The uncorrelator and mixer 409 are then configured to apply a mixing procedure to generate a time-frequency space audio signal 410. y(b,t)=M(b,n)x(b,t)+M r (b,n)x D (b,t)

[0166] Although not explicitly stated in the formula, the processing matrix may be linearly interpolated between frames n such that the matrix steps from M(b,n-1) to M(b,n) at each time index of the time-frequency signal. The interpolation rate may depend on whether the start is detected (fast interpolation) or not (normal interpolation). The time-frequency spatial audio signal y(b,t)410 is then output.

[0167] In some embodiments, the spatial combiner 207 includes an inverse filter bank 411 configured to receive a spatial-time-frequency domain audio signal 410 and apply an inverse transform corresponding to the transform applied by the forward filter bank 401 to generate a time-domain spatial output signal 110. Thus, the output of the inverse filter bank 411 may be a spatial output signal 110, which may be, for example, a binaural audio signal for headphone listening.

[0168] The covariance matrix-based rendering scheme shown in Figure 4 is merely one illustrative configuration, and other configurations are implemented in several further embodiments. For example, in some embodiments, an audio signal may be split into directional and omnidirectional parts (or directional and omnidirectional sounds) in the frequency band based on a ratio parameter, the directional parts may then be positioned on virtual loudspeakers using amplitude panning, the omnidirectional parts may be distributed to all loudspeakers and decorrelated, the processed directional and omnidirectional parts may then be added together, and finally, each virtual loudspeaker is processed with an HRTF to obtain a binaural output. This procedure is described in more detail in relation to the DirAC rendering scheme described in Laitinen, MV, & Pulkki, V. (2009, October). Binaural reproduction for directional audio coding. In 2009 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (pp. 337-340). In this disclosure, the coding quality metric may be used to control the time averaging of audio processing operations, in particular the processing gain that positions the directional portion to a virtual loudspeaker.

[0169] A similar method can be applied to rendering multi-channel loudspeaker signals (e.g., 5.1). In this case, virtual loudspeakers replace the actual loudspeakers, and the combined loudspeaker signals are output as multi-channel loudspeaker signals without the application of binauralization.

[0170] Regarding Figure 5, a flowchart illustrating the operation of the example spatial synthesizer shown in Figure 4 is presented.

[0171] For example, Figure 5 shows the operation of receiving the decoded transport audio signal in step 501.

[0172] In addition, the decoded transport audio signal is then subjected to a time-frequency domain conversion by step 503, as shown in Figure 5.

[0173] The decrypted spatial metadata is received by step 502 as shown in Figure 5.

[0174] Next, the covariance matrix is ​​determined based on the spatial metadata and the decoded transport audio signal, as shown in Figure 5, by step 505.

[0175] The coding quality metric is also received by step 504, as shown in Figure 5.

[0176] The time averaging of the covariance matrix based on the coding quality metric is shown in Figure 5 by step 507.

[0177] The determination of the processing matrix based on the time-mean covariance matrix and the time-frequency domain transport audio signal is shown in Figure 5 by step 509.

[0178] Next, decorrelation and mixing of time-frequency domain transport audio signals based on the processing matrix is ​​shown in Figure 5 by step 511.

[0179] Next, the application of the inverse time-frequency domain transform to generate the spatial audio signal is shown in Figure 5 by step 513.

[0180] Then, the spatial audio signal is output by step 515 as shown in Figure 5.

[0181] In some embodiments, the frequency-independent coding quality metric Ξ(n) is, alternatively, the target spatial directional quantization resolution η per frequency band k and time frame n. target (k,n) metric and achieved spatial directional quantization resolution η actual These can be defined in relation to the (k,n) metric. These metrics directly represent the minimum precision of quantization in the spatial direction, expressed as distance on a sphere, i.e., spatial angle. These metrics can be predetermined for each allowable bit budget; for example, 11 bits can allow a minimum precision of 4°, and 5 bits can allow a minimum precision of 36°. To obtain an equivalent coding quality metric Ξ(n), such a method first obtains the per-band metric.

[0182]

number

number

[0183] In such embodiments, K is the total number of frequency bands. This combination can be implemented similarly in various other ways, for example, using a maximum or minimum function, a median function, a summation function, or with averaged weights. In addition, in some embodiments, the band-by-band metric or frequency-independent metric can have logarithmic or exponential functions applied to its values. It is possible that there are individual values ​​of spatial resolution for each time-frequency (TF) tile within a subband. In such embodiments, η actual (k,n) can be obtained by averaging the values ​​across TF tiles.

[0184] As described above, in some embodiments, the coding quality metric Ξ(n) can be set to a value based on information other than that which is normally used. For example, the coding metric can be based on the number of bits used, or parameters related to the quantization resolution of the coded metadata, such as the spatial quantization resolution, or the ratio between the quantization resolutions used to code the metadata. One such case is when there is prior knowledge that a particular coding method always achieves the target quality. In such a case, Ξ(k,n)=1 is always the case. Another such case is η actual (k,n) is η target This is the case when the performance is considered sufficiently good for each frequency band k, regardless of the values ​​of (k,n). This relates, for example, to use cases with very low bitrates where the resulting output may not be of good quality.

[0185] As discussed above, the (frequency-independent) coding quality metric Ξ(n) can be defined, for example, as the relationship between the target bit budget and the actual bits used in spatial directional coding. In further embodiments, the metric may take into account at least one of the following: the direct-to-total energy ratio for each affected direction, and the direct-to-total energy ratio* for each affected direction.

[0186] In some embodiments, the coding quality metric may be (at least partially) frequency-dependent. This frequency dependence may be related to the TF resolution of the spatial metadata quantizer, at least. For example, using the embodiments described above, α and β may be determined as frequency-dependent values ​​based on the frequency-dependent Ξ(k,n). In these embodiments, the remainder of the rendering procedure can be performed as presented above.

[0187] The exemplary embodiments presented above adjust time smoothing based on an encoding quality metric. In some alternative embodiments, rendering can also be adjusted in other ways (instead of or in addition to time smoothing). Several examples are presented below.

[0188] In some embodiments, the direct-to-total energy ratio r(k,n) can be adjusted based on the coding quality metric Ξ(n). The idea is to reduce the direct-to-total energy ratio r(k,n) when the coding quality is lower than 1. As a result, less sound is rendered as directional, and thus directional variation is less perceived. This can be implemented, for example, as follows: First, the direct-to-total energy ratio is, for example,

[0189]

number

[0190] In some embodiments, the direct sound rendering pattern can be modified based on the coding quality metric Ξ(n). For example, when Ξ(n) is less than 1, a broader source-compatible HRTF may be used instead of a point-like HRTF (e.g., based on a linear pattern). This effectively reduces the accuracy of perceived directivity, thereby mitigating artifacts caused by directional variations.

[0191] In some embodiments, the orientation can be smoothed over time based on an encoding quality metric Ξ(n). For example, if Ξ(n) = 1, the original orientation can be used, but if Ξ(n) < 1, smoothing over time can be performed on the orientation. This can be done, for example, by converting the orientation to a Cartesian coordinate vector and then smoothing the orientation over time using an IIR filter.

[0192] With respect to Figure 6, the illustrative electronic device may be used as one of the device components of the system described above. The device may be any suitable electronic device or apparatus. For example, in some embodiments, device 1400 may be a mobile device, user equipment, tablet computer, computer, audio playback device, etc. The device may be configured to implement, for example, an encoder / analyzer section and / or decoder section as shown in Figure 1, or any functional block as described above.

[0193] In some embodiments, the device 1400 includes at least one processor or central processing unit 1407. The processor 1407 can be configured to execute various program codes, such as those described herein.

[0194] In some embodiments, the device 1400 includes at least one memory 1411. In some embodiments, at least one processor 1407 is coupled to the memory 1411. The memory 1411 can be any suitable storage means. In some embodiments, the memory 1411 includes a program code section for storing program code that can be implemented by the processor 1407. Furthermore, in some embodiments, the memory 1411 may further include a stored data section for storing data, for example, data that has been processed or is to be processed according to embodiments such as those described herein. The program code to be implemented stored in the program code section, and the data stored in the stored data section, can be retrieved by the processor 1407 via the memory-processor coupling whenever necessary.

[0195] In some embodiments, device 1400 includes a user interface 1405. In some embodiments, the user interface 1405 may be coupled to a processor 1407. In some embodiments, the processor 1407 can control the operation of the user interface 1405 and receive input from the user interface 1405. In some embodiments, the user interface 1405 may allow a user to input commands to device 1400, for example, via a keypad. In some embodiments, the user interface 1405 may allow a user to retrieve information from device 1400. For example, the user interface 1405 may include a display configured to show information from device 1400 to the user. In some embodiments, the user interface 1405 may include a touchscreen or touch interface that has the ability to both allow information to be input to device 1400 and to further display the information to the user of device 1400. In some embodiments, the user interface 1405 may be a user interface for communication.

[0196] In some embodiments, device 1400 includes an input / output port 1409. In some embodiments, the input / output port 1409 includes a transceiver. The transceiver in such embodiments may be coupled to a processor 1407 and configured to enable communication with other devices or electronic devices, for example, via a wireless communication network. The transceiver, or any suitable transceiver, or transmitter and / or receiver means, is configured in some embodiments to communicate with other electronic devices or devices via a wire or wired coupling.

[0197] The transceiver can communicate with further devices by any suitable known communication protocol. For example, in some embodiments, the transceiver can use a suitable radio access architecture based on Long-Term Evolution Advanced (LTE Advanced, LTE-A) or New Radio (NR) (sometimes called 5G), Universal Mobile Communications System (UMTS) Radio Access Network (UTRAN or E-UTRAN), Long-Term Evolution (LTE, same as E-UTRA), 2G Network (Legacy Network Technology), Wireless Local Area Network (WLAN or Wi-Fi), Global Interoperability Microwave Access (WiMAX), Bluetooth®, Personal Communication Services (PCS), ZigBee®, Wideband Code Division Multiple Access (WCDMA), Systems using Ultra-Wideband (UWB) technology, Sensor Networks, Mobile Ad Hoc Networks (MANET), Cellular Internet of Things (IoT) RAN and Internet Protocol Multimedia Subsystem (IMS), any other suitable option, and / or any combination thereof.

[0198] The transceiver input / output port 1409 can be configured to receive signals.

[0199] In some embodiments, device 1400 may be used as at least part of a synthesis device. The input / output port 1409 may be coupled to headphones (which may be head-tracking headphones or non-tracking headphones) or similar, and a loudspeaker.

[0200] In general, various embodiments of the present invention may be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. For example, some embodiments may be implemented in hardware, while others may be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device, but the present invention is not limited thereto. Various embodiments of the present invention may be illustrated and described using block diagrams, flowcharts, or any other graphic representation, but it will be understood that these blocks, apparatus, systems, techniques, or methods described herein may, in non-limiting examples, be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers, or other computing devices, or any combination thereof.

[0201] Embodiments of the present invention may be implemented by computer software executable by a data processor in a mobile device, such as within a processor entity, by hardware, or by a combination of software and hardware. Furthermore, it should be noted that any block of the logic flow as shown in the figure may represent a program step, or an interconnected set of logic circuits, blocks, and functions, or a combination of program steps and logic circuits, blocks, and functions. The software may be stored on a memory chip, or a memory block implemented within the processor, on a magnetic medium such as a hard disk or floppy disk, and on a physical medium such as an optical medium such as a DVD and its data variants, or a CD.

[0202] The memory can be of any type suitable for the local technology environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The data processor can be of any type suitable for the local technology environment and may further include, in non-limiting examples, one or more of general-purpose computers, dedicated computers, microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), gate-level circuits, and processors based on multi-core processor architectures.

[0203] Embodiments of the present invention can be put into practice in various components such as integrated circuit modules. Designing integrated circuits is generally a highly automated process. Complex and powerful software tools are available to translate logic-level designs into semiconductor circuit designs ready for etching and forming on semiconductor substrates.

[0204] Programs such as those offered by Synopsys, Inc. in Mountain View, California, and Cadence Design in San Jose, California, use well-established design rules and a library of pre-stored design modules to automatically route conductors and place components on a semiconductor chip. Once the semiconductor circuit design is complete, the resulting design in a standardized electronic format (e.g., Opus, GDSII, etc.) can be sent to a semiconductor manufacturing facility or "fab" for production.

[0205] The foregoing description provides a complete and informative explanation of exemplary embodiments of the invention, in illustrative and non-limiting terms. However, various modifications and adaptations will become apparent to those skilled in the art, in conjunction with the accompanying drawings and claims, considering the foregoing description. Nevertheless, all such and similar modifications of the teachings of the invention still fall within the scope of the invention as defined in the accompanying claims.

Claims

1. Obtaining a bitstream containing encoded spatial metadata and encoded transport audio signals, Decoding the transport audio signal from the transport audio signal encoded in the bitstream, Decoding spatial metadata from the spatial metadata encoded in the bitstream, Generating an encoded metric, To generate a spatial audio signal from the transport audio signal based on the encoding metric and the spatial metadata. Includes means for, The means for generating a spatial audio signal from the transport audio signal based on the encoding metric and the spatial metadata, A covariance matrix is ​​generated from the transport audio signal and the spatial metadata based on the encoding metric, The process matrix is ​​generated based on the aforementioned covariance matrix, To generate the spatial audio signal, the transport audio signal is decorrelated and / or mixed based on the processing matrix. A device intended for that purpose.

2. The apparatus according to claim 1, wherein the means for generating a covariance matrix from the transport audio signal and the spatial metadata based on the coding metric further comprises modifying at least the energy ratio from the spatial metadata based on the coding metric, and the spatial audio signal is generated from the transport audio signal based on the modified energy ratio and the spatial metadata.

3. The apparatus according to claim 1, wherein the means for generating a covariance matrix from the transport audio signal and the spatial metadata based on the coding metric is for positioning a directional sound in a direction determined by the spatial metadata, and the width of the directional sound is based on the coding metric.

4. The means for generating the covariance matrix from the transport audio signal and the spatial metadata based on the encoding metric, From the transport audio signal and the spatial metadata, an input covariance matrix representing the transport audio signal, and To generate a target covariance matrix representing the aforementioned spatial audio signal, Based on the coding metric, the covariance matrix is ​​generated from the input covariance matrix and the target covariance matrix. It is for that purpose. The apparatus according to claim 1.

5. The apparatus according to claim 4, wherein the means for generating a covariance matrix from the transport audio signal and the spatial metadata based on the coding metric is for generating the input covariance matrix by measuring the transport audio signal in the time-frequency domain.

6. The apparatus according to claim 4, wherein the means for generating a covariance matrix from the transport audio signal and the spatial metadata based on the coding metric is for generating the target covariance matrix based on the spatial metadata and the transport audio signal energy.

7. The apparatus according to claim 2, wherein the means for generating a covariance matrix from the transport audio signal and the spatial metadata based on the coding metric is for generating the covariance matrix based on the modified energy ratio.

8. The apparatus according to any one of claims 1 to 7, wherein the means for generating the coding metric is for generating the coding metric based on the quality of representation of the spatial metadata.

9. The apparatus according to any one of claims 1 to 7, wherein the means for generating the coded metric comprises coded spatial metadata and means for generating the coded metric from the spatial metadata.

10. The means for generating encoded spatial metadata and encoded metrics from the spatial metadata are, Determining a first parameter that indicates the number of bits intended or allocated to encode spatial parameters for a frame, Determining a second parameter that indicates the number of bits used after encoding the spatial parameters has been performed on the frame, The coding metric is generated as the ratio between the first parameter and the second parameter. The apparatus according to claim 9, which is for the purpose of

11. The apparatus according to claim 10, wherein the spatial parameter is a directional index representing a quantized directional parameter value.

12. The means for generating the coding metric is, The quantization resolution of the aforementioned space metadata, and The ratio between at least two quantization resolutions of the spatial metadata The apparatus according to any one of claims 1 to 7, for generating the coding metric based on at least one of the following.

13. Obtaining a bitstream containing encoded spatial metadata and encoded transport audio signals, Decoding the transport audio signal from the transport audio signal encoded in the bitstream, Decoding spatial metadata from the spatial metadata encoded in the bitstream, Generating an encoded metric, To generate a spatial audio signal from the transport audio signal based on the encoding metric and the spatial metadata. Includes, The step of generating a spatial audio signal from the transport audio signal based on the encoding metric and the spatial metadata is: A covariance matrix is ​​generated from the transport audio signal and the spatial metadata based on the encoding metric, The process matrix is ​​generated based on the aforementioned covariance matrix, To generate the spatial audio signal, the transport audio signal is decorrelated and / or mixed based on the processing matrix. It is for that purpose. method.

Images