Low-complexity low-bitrate 6DOF HOA rendering method and apparatus

By generating position-interpolated high-order ambisonic sources during rendering, the method addresses the high computational and bandwidth demands of spatial audio capture, enabling efficient 6DoF rendering with reduced complexity and bandwidth, suitable for low-end devices.

JP7579311B2Active Publication Date: 2024-11-07NOKIA TECHNOLOGIES OY
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2022165971
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-10-18
Filing Date
2022-10-17
Publication Date
2024-11-07
Estimated Expiration
2042-10-17

AI Technical Summary

Technical Problem

Existing spatial audio capture technologies face challenges in efficiently capturing and rendering high-order ambisonic signals due to the need for high computational power and bandwidth, particularly in systems requiring multiple microphone arrays to handle both short and long wavelengths, which is difficult to achieve with a single array.

Method used

The proposed method generates position-interpolated high-order ambisonic sources during rendering metadata creation, reducing the need for multiple audio sources by interpolating spatial metadata and audio signals, allowing for 6 degrees of freedom (6DoF) rendering with a single source, thereby lowering computational complexity and bandwidth requirements.

Benefits of technology

This approach reduces MPHOA rendering bandwidth by up to 1/3 and makes low-end equipment capable of handling complex 6DoF audio scenes, while maintaining sound quality, by shifting computational complexity to content processing and storage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007579311000006
    Figure 0007579311000006
  • Figure 0007579311000007
    Figure 0007579311000007
  • Figure 0007579311000008
    Figure 0007579311000008
Patent Text Reader

Abstract

To provide a device and method for generating an immersive audio scene.SOLUTION: The method acquires two or more audio scene based sources AS1 to AS5 including a spacial parameter and an audio signal, related to a position in the audio scene, determines the position for rendering associated with the audio scene based source, generates a spacial audio parameter based on the spacial parameter related to the audio scene based source, generates a sound source signal based on the audio signal of an audio source, generates information related to a relation between the spacial audio parameter and the audio signal related to the audio scene based source, and selects a sound source based on a preference of a renderer.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present application relates to an apparatus and method for audio rendering with spatial metadata interpolation for an audio scene containing high-order Ambisonics sources at known positions for a user with six degrees of freedom. [Background technology]

[0002] Spatial audio capture approaches attempt to capture the audio environment such that it is perceptually reproduced in a manner that is effective for the listener, and further, the listener can move and / or rotate within the reproduced audio environment. For example, in some systems (three degrees of freedom - 3DoF), the listener can rotate his / her head and the rendered audio signal reflects this rotational motion. In some systems (three degrees of freedom plus - 3DoF+), the listener can simultaneously rotate his / her head and "move" slightly within the environment, and in other systems (six degrees of freedom - 6DoF), the listener can move freely within the environment and rotate his / her head.

[0003] Linear spatial audio capture refers to an audio capture method in which the processing does not adapt to the characteristics of the captured audio. Instead, the output is a predetermined linear combination of the captured audio signals.

[0004] To record spatial sound linearly at a certain position in the recording space, a high-end microphone array is required. One such array is the spherical 32-microphone "Eigenmike." From this microphone array, a High Order Ambisonics (HOA) signal can be obtained and used for rendering. Using the HOA signal, spatial audio can be rendered with sufficient separation of sounds coming from different directions in the appropriate audible bandwidth.

[0005] In linear spatial audio capture technology, the performance of the microphone array is an issue: short wavelengths (high frequency audio signals) require small microphone spacing, while long wavelengths (low frequency audio signals) require large array size, and it is difficult to meet both conditions with a single microphone array.

[0006] Implementing linear spatial audio capture in a capture device results in spatial audio being captured at only one position.

[0007] Parametric spatial audio capture is a system that estimates perceptually relevant parameters based on an audio signal captured by a microphone, and then synthesizes spatial sound based on these parameters and the audio signal. Analysis and synthesis is usually performed in a frequency range that is close to the spatial resolution of human hearing.

[0008] MPEG-I immersive audio is currently being standardized. It is assumed that MPEG-I immersive audio will receive three types of audio signal formats: object, channel, and HOA. Higher-Order Ambisonics (HOA) sources, one of the signal formats adopted by MPEG-I, are useful when capturing object audio is not possible or is too complex. HOA audio can be created from live capture or synthesized from a virtual scene containing many objects. Multiple HOA sources can be used to represent a scene, which can be moved in six degrees of freedom. In general, for scene-based audio capture, one or more HOA sources are created by capturing an audio scene with suitable microphones (e.g., a microphone array).

[0009] Rendering is the process of processing a captured audio signal (or a transport audio signal derived from the captured audio signal) and parameters to produce an output suitable for presentation to a listener via an appropriate audio transducer, e.g., headphones or loudspeakers. Summary of the Invention [Problem to be solved by the invention]

[0010] SUMMARY OF THE PRESENT EMBODIMENTS The present invention relates to an embodiment that addresses problems associated with the prior art. [Means for solving the problem]

[0011] According to a first aspect, there is provided an apparatus for generating an immersive audio scene, the apparatus comprising: acquiring two or more audio scene based sources, the two or more audio scene based sources being associated with one or more positions in an audio scene, each audio scene based source comprising at least one spatial parameter and at least one audio signal; determining at least one position associated with at least one of the acquired two or more audio scene based sources, the at least one position being determined for rendering; and generating at least one audio source based on the determined at least one position, wherein means configured to generate the at least one audio source comprises: the at least one audio source signal for the at least one audio source based on the at least one audio signal of the at least one audio scene-based source obtained in association with the determined at least one location; and generating information regarding a relationship between the generated at least one spatial audio parameter and the at least one audio signal associated with the at least one audio scene-based source obtained in association with the determined at least one location, wherein the generated at least one audio source signal is selected based on a preference of a renderer.

[0012] The means configured to determine at least one position associated with at least one of the obtained two or more audio scene-based sources may be configured to obtain the at least one position from the at least one further device, and the means may be further configured to: transmit information to the at least one further device; and, when selecting the two or more audio scene-based sources, output at least one spatial parameter and at least one audio signal of the selected two or more sources; and, when selecting the at least one audio source, output at least one spatial audio parameter of the audio source and the at least one audio source signal.

[0013] The means configured to select the two or more audio scene-based sources or the at least one audio source based on a position from the at least one further device may be configured to select the two or more audio scene-based sources or the at least one audio source based on at least one of a bandwidth of a transmission or storage channel between the device and the further device and a computing capability of the further device.

[0014] The means configured to generate the at least one audio source based on the determined at least one position may be configured to determine a position of the at least one audio source based on the determined at least one position from the at least one further device.

[0015] The means configured to generate at least one audio source based on the determined at least one position may be configured to: select or define a group of audio scene based sources within the two or more audio scene based sources; generate at least one spatial audio parameter based on a combination of the two or more audio scene based sources and at least one spatial parameter from the selected or defined group of audio scene based sources within the two or more audio scene based sources; and generate at least one audio source signal based on a combination of the two or more audio scene based sources and at least one audio signal from the selected or defined group of audio scene based sources within the two or more audio scene based sources.

[0016] The means configured to acquire the two or more audio scene-based sources may be configured to acquire at least two audio signals from microphones positioned in the audio scene, and to analyze the at least two audio signals to identify the two or more audio scene-based sources and at least one spatial parameter and at least one audio signal associated with each of the two or more audio scene-based sources.

[0017] The means configured to obtain the two or more audio scene-based sources may be configured to receive or synthesize the two or more audio scene-based sources.

[0018] Two or more of the audio scene-based sources may be higher-order Ambisonics sources.

[0019] The at least one audio source generated based on the determined at least one position may be a position-interpolated high-order Ambisonics source.

[0020] According to a second aspect, an apparatus for spatial audio signal rendering is provided, comprising means configured to: obtain information regarding a relation between the generated at least one spatial audio parameter and at least one audio signal associated with at least one of the obtained two or more audio scene based sources and the generated at least one audio source; obtain a user position value and a user orientation value; request a selection of the generated at least one audio source and / or at least two of the two or more audio scene based sources based on the user position value; obtain at least one rendering source spatial parameter based on the request; obtain at least one rendering source audio signal based on the request; and generate at least one output audio signal based on the user orientation value, the at least one rendering source spatial parameter and the at least one rendering source audio signal.

[0021] The means configured to request selection of the generated at least one audio source and / or at least two of the two or more audio scene-based sources based on a user position value may be configured to determine at least one of the bandwidth of a transmission or storage channel between the device from which the at least one rendering source spatial parameter and the at least one rendering source audio signal are obtained and the further device, and the computing capacity of the device, and to select the at least one audio source or at least two of the two or more audio scene-based sources based on the bandwidth or the computing capacity of the transmission or storage channel.

[0022] According to a third aspect, there is provided a method for an apparatus for generating an immersive audio scene, the method comprising: acquiring two or more audio scene based sources, the two or more audio scene based sources being associated with one or more positions in an audio scene, each audio scene based source including at least one spatial parameter and at least one audio signal; determining at least one position associated with at least one of the acquired two or more audio scene based sources, the at least one position being determined for rendering; and generating at least one audio source based on the determined at least one position, the generating at least one audio source being associated with the determined at least one position. and generating at least one spatial audio parameter based on at least one spatial parameter associated with at least one of the two or more acquired audio scene-based sources, in association with the determined at least one location, and generating at least one audio source signal for the at least one audio source based on at least one audio signal of the two or more acquired audio scene-based sources, wherein the generated at least one audio source is selected based on a preference of a renderer.

[0023] Determining at least one location associated with at least one of the obtained two or more audio scene-based sources includes obtaining the at least one location from the at least one further device, and the method may further include transmitting information to the at least one further device; and when selecting the two or more audio scene-based sources, outputting at least one spatial parameter and at least one audio signal of the selected two or more sources; and when selecting the at least one audio source, outputting the at least one spatial audio parameter of the audio source and the at least one audio source signal.

[0024] Selecting the two or more audio scene-based sources or the at least one audio source based on a position from the at least one further device includes selecting the two or more audio scene-based sources or the at least one audio source based on at least one of a bandwidth of a transmission or storage channel between the device and the further device and a computing capability of the further device.

[0025] Generating the at least one audio source based on the determined at least one location may include determining a location of the at least one audio source based on the determined at least one location from the at least one further device.

[0026] Generating at least one audio source based on the determined at least one location may include selecting or defining a group of audio scene based sources from the two or more audio scene based sources, generating at least one spatial audio parameter based on a combination of the two or more audio scene based sources and at least one spatial parameter from the selection of the two or more audio scene based sources or the defined group of audio scene based sources, and generating at least one audio source signal based on a combination of the two or more audio scene based sources and at least one audio signal from the selection of the two or more audio scene based sources or the defined group of audio scene based sources.

[0027] Acquiring the two or more audio scene-based sources may include acquiring at least two audio signals from microphones located in the audio scene, and analyzing the at least two audio signals to identify the two or more audio scene-based sources, as well as at least one spatial parameter and at least one audio signal associated with each of the two or more audio scene-based sources.

[0028] Obtaining the two or more audio scene-based sources may include receiving or synthesizing the two or more audio scene-based sources.

[0029] Two or more of the audio scene-based sources may be higher-order Ambisonics sources.

[0030] The at least one audio source generated based on the determined at least one position may be a position-interpolated high-order Ambisonics source.

[0031] According to a fourth aspect, there is provided a method for an apparatus for spatial audio signal rendering, the method comprising: obtaining information regarding a relationship between the generated at least one spatial audio parameter and at least one of the acquired two or more audio scene based sources and at least one audio signal related to the generated at least one audio source; obtaining a user position value and a user direction value; requesting selection of the generated at least one audio source and / or at least two of the two or more audio scene based sources based on the user position value; obtaining at least one rendering source spatial parameter based on the request; obtaining at least one rendering source audio signal based on the request; and generating at least one output audio signal based on the user direction value, the at least one rendering source spatial parameter and the at least one rendering source audio signal.

[0032] Requesting selection of the generated at least one audio source and / or at least two of the two or more audio scene-based sources based on the user position value may include determining at least one of the bandwidth of a transmission or storage channel between the device from which the at least one rendering source spatial parameter and the at least one rendering source audio signal are obtained and the further device, and computing capabilities of the device, and selecting the at least one audio source or at least two of the two or more audio scene-based sources based on the bandwidth of the transmission or storage channel or the computing capabilities.

[0033] According to a fifth aspect, there is provided an apparatus for generating an immersive audio scene, the apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code causing the apparatus, using the at least one processor, to at least: acquire two or more audio scene based sources, the two or more audio scene based sources being associated with one or more positions in the audio scene, each audio scene based source including at least one spatial parameter and at least one audio signal; determine at least one position associated with at least one of the acquired two or more audio scene based sources, the at least one position being determined for rendering; and generate the at least one audio source based on the determined at least one position. the means configured to generate the at least one audio source is configured to generate at least one spatial audio parameter based on at least one spatial parameter associated with at least one of the two or more acquired audio scene-based sources, in association with the determined at least one location, and generate a signal for the at least one audio source based on an audio signal of the at least one acquired audio scene-based source, in association with the determined at least one location; and generating information regarding a relationship between the generated at least one spatial audio parameter and the at least one audio signal associated with at least one of the two or more acquired audio scene-based sources, where the generated at least one audio source is selected based on a preference of the renderer.

[0034] The apparatus adapted to determine at least one position associated with at least one of the obtained two or more audio scene-based sources may be adapted to obtain the at least one position from the at least one further apparatus, and the apparatus may be further adapted to transmit information to the at least one further apparatus, and to output, upon selection of the two or more audio scene-based sources, at least one spatial parameter and at least one audio signal of the selected two or more sources, and to output, upon selection of the at least one audio source, at least one spatial audio parameter and at least one audio source signal of the audio source.

[0035] The apparatus adapted to select two or more audio scene-based sources or the at least one audio source based on a position from the at least one further apparatus may be adapted to select the two or more audio scene-based sources or the at least one audio source based on at least one of a bandwidth of a transmission or storage channel between the apparatus and the further apparatus and a computing capability of the further apparatus.

[0036] The device adapted to generate the at least one audio source based on the determined at least one position may be adapted to determine a position of the at least one audio source based on the determined at least one position from the at least one further device.

[0037] The apparatus configured to generate at least one audio source based on the determined at least one position may be configured to: select or define a group of audio scene based sources among the two or more audio scene based sources; generate at least one spatial audio parameter based on a combination of the two or more audio scene based sources and at least one spatial parameter from the selected or defined group of audio scene based sources among the two or more audio scene based sources; and generate at least one audio source signal based on a combination of the two or more audio scene based sources and at least one audio signal from the selected or defined group of audio scene based sources among the two or more audio scene based sources.

[0038] An apparatus adapted to acquire two or more audio scene-based sources may be adapted to acquire at least two audio signals from microphones positioned in the audio scene, and analyze the at least two audio signals to identify the two or more audio scene-based sources, as well as at least one spatial parameter and at least one audio signal associated with each of the two or more audio scene-based sources.

[0039] A device adapted to acquire two or more audio scene-based sources may be adapted to receive or synthesize two or more audio scene-based sources.

[0040] Two or more of the audio scene-based sources may be higher-order Ambisonics sources.

[0041] The at least one audio source generated based on the determined at least one position may be a position-interpolated high-order Ambisonics source.

[0042] According to a sixth aspect, there is provided an apparatus for spatial audio signal rendering, comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to cause the apparatus to at least: obtain information regarding a relationship between the generated at least one spatial audio parameter and at least one audio signal associated with at least one of the acquired two or more audio scene based sources and the generated at least one audio source; obtain a user position value and a user orientation value; request selection of the generated at least one audio source and / or at least two of the two or more audio scene based sources based on the user position value; obtain at least one rendering source spatial parameter based on the request; obtain at least one rendering source audio signal based on the request; and generate at least one output audio signal based on the user orientation value, the at least one rendering source spatial parameter, and the at least one rendering source audio signal.

[0043] The device requested to select the generated at least one audio source and / or at least two of the two or more audio scene-based sources based on the user position value may further be configured to determine at least one of the bandwidth of a transmission or storage channel between the device from which the at least one rendering source spatial parameter and the at least one rendering source audio signal are obtained and the further device and the computing capacity of the device, and to select the at least one audio source or at least two of the two or more audio scene-based sources based on the bandwidth of the transmission or storage channel or the computing capacity.

[0044] According to a seventh aspect, there is provided an apparatus for generating a spatialized audio output based on a user position, the apparatus comprising: means for acquiring two or more audio scene based sources, the two or more audio scene based sources being associated with one or more positions in an audio scene, each audio scene based source comprising at least one spatial parameter and at least one audio signal; means for determining at least one position associated with at least one of the acquired two or more audio scene based sources, the at least one position being determined for rendering; and means for generating at least one audio source based on the determined at least one position, the means for generating the at least one audio source based on the determined at least one position. and means for generating at least one spatial audio parameter based on at least one spatial parameter associated with at least one of the two or more acquired audio scene-based sources in association with the determined at least one position, and means for generating at least one audio source signal for the at least one audio source based on at least one audio signal of the two or more acquired audio scene-based sources in association with the determined at least one position, and means for generating information regarding a relationship between the generated at least one spatial audio parameter and the at least one audio signal associated with at least one of the two or more acquired audio scene-based sources, where the generated at least one audio source is selected based on a preference of a renderer.

[0045] According to an eighth aspect, there is provided an apparatus for generating a spatialized audio output based on a user position, comprising: means for obtaining information regarding a relationship between the generated at least one spatial audio parameter, the obtained two or more audio scene based sources, and at least one audio signal associated with at least one of the generated at least one audio source; means for obtaining a user position value and a user direction value; means for requesting selection of the generated at least one audio source and / or at least two of the two or more audio scene based sources based on the user position value; means for obtaining at least one rendering source spatial parameter based on the request; means for obtaining at least one rendering source audio signal based on the request; and means for generating at least one output audio signal based on the user direction value, the at least one rendering source spatial parameter, and the at least one rendering source audio signal.

[0046] According to a ninth aspect, an apparatus is provided with a method for generating at least one audio source based on at least one of the acquired audio scene-based sources, the two or more audio scene-based sources being associated with one or more positions in an audio scene, each audio scene-based source including at least one spatial parameter and at least one audio signal; determining at least one position associated with at least one of the acquired two or more audio scene-based sources, the at least one position being determined for rendering; and generating at least one audio source based on the determined at least one position, the generating at least one audio source being associated with at least one of the acquired two or more audio scene-based sources, the at least one position being determined for rendering. and generating at least one audio signal for the at least one audio source based on at least one audio signal of the two or more acquired audio scene based sources in association with the determined at least one location; and generating information regarding a relationship between the generated at least one spatial audio parameter and the at least one audio signal associated with at least one of the two or more acquired audio scene based sources, wherein the generated at least one audio source is selected based on a preference of a renderer.

[0047] According to a ninth aspect, there is provided a computer program comprising instructions [or a computer readable medium comprising program instructions] to cause an apparatus to at least obtain information regarding a relationship between the generated at least one spatial audio parameter and at least one of the two or more obtained audio scene based sources and at least one audio signal related to the generated at least one audio source; obtain a user position value and a user orientation value; request selection of the generated at least one audio source and / or at least two of the two or more audio scene based sources based on the user position value; obtain at least one rendering source spatial parameter based on the request; obtain at least one rendering source audio signal based on the request; and generate at least one output audio signal based on the user orientation value, the at least one rendering source spatial parameter and the at least one rendering source audio signal.

[0048] According to an eleventh aspect, a method for generating at least one audio source includes: acquiring, on an apparatus, two or more audio scene based sources, the two or more audio scene based sources being associated with one or more positions in an audio scene, each audio scene based source including at least one spatial parameter and at least one audio signal; determining at least one position associated with at least one of the acquired two or more audio scene based sources, the at least one position being determined for rendering; and generating at least one audio source based on the determined at least one position, the generating at least one audio source including ... a first audio signal associated with the at least one audio scene based source signal for the at least one audio source based on at least one audio signal of the two or more audio scene based sources obtained in association with the determined at least one location; and generating information regarding a relationship between the generated at least one spatial audio parameter and the at least one audio signal associated with at least one of the two or more audio scene based sources obtained, where the generated at least one audio source is selected based on a preference of a renderer.

[0049] According to a twelfth aspect, there is provided a non-transitory computer-readable medium comprising program instructions to cause an apparatus to at least obtain information regarding a relationship between the generated at least one spatial audio parameter, at least one of the two or more acquired audio scene-based sources and at least one audio signal associated with the generated at least one audio source, obtain a user position value and a user orientation value, request selection of the generated at least one audio source and / or at least two of the two or more audio scene-based sources based on the user position value, obtain at least one rendering source spatial parameter based on the request, obtain at least one rendering source audio signal based on the request, and generate at least one output audio signal based on the user orientation value, the at least one rendering source spatial parameter, and the at least one rendering source audio signal.

[0050] According to a thirteenth aspect, there is provided an audio system comprising: an acquisition circuit configured to acquire two or more audio scene based sources, the two or more audio scene based sources being associated with one or more positions in an audio scene, each audio scene based source including at least one spatial parameter and at least one audio signal; a determination circuit configured to determine at least one position associated with at least one of the acquired two or more audio scene based sources, the at least one position being determined for rendering; and a generation circuit configured to generate the at least one audio source based on the determined at least one position, the generation circuit configured to generate the at least one audio source based on the two or more acquired audio scene based sources associated with the determined at least one position. and generating at least one audio source signal for the at least one audio source based on at least one audio signal of the two or more acquired audio scene-based sources in association with the determined at least one location; and a generating circuit configured to generate information regarding a relationship between the generated at least one spatial audio parameter and the at least one audio signal associated with at least one of the acquired two or more audio scene-based sources, wherein the generated at least one audio source is selected based on a preference of a renderer.

[0051] According to a fourteenth aspect, there is provided an apparatus comprising: an acquisition circuit configured to acquire information regarding a relationship between the generated at least one spatial audio parameter, at least one of the acquired two or more audio scene-based sources, and at least one audio signal related to the generated at least one audio source; an acquisition circuit configured to acquire a user position value and a user orientation value and to request selection of the generated at least one audio source and / or at least two of the two or more audio scene-based sources based on the user position value; an acquisition circuit configured to acquire the at least one rendering source spatial parameter based on the request; an acquisition circuit configured to acquire the at least one rendering source audio signal based on the request; and a generation circuit configured to generate at least one output audio signal based on the user orientation value, the at least one rendering source spatial parameter, and the at least one rendering source audio signal.

[0052] According to a fifteenth aspect, a method includes: acquiring, on an apparatus, at least two or more audio scene based sources associated with one or more positions in an audio scene, each audio scene based source including at least one spatial parameter and at least one audio signal; determining at least one position associated with at least one of the acquired two or more audio scene based sources, the at least one position being determined for rendering; and generating at least one audio source based on the determined at least one position, the generating of the at least one audio source comprising: acquiring, on an apparatus, two or more audio scene based sources associated with the acquired two or more audio scene based sources, the at least one position being determined for rendering; and generating at least one audio source signal for the at least one audio source based on at least one audio signal of the two or more audio scene-based sources obtained in association with the determined at least one location; and generating information regarding a relationship between the generated at least one spatial audio parameter and the at least one audio signal associated with at least one of the two or more acquired audio scene-based sources, where the generated at least one audio source is selected based on a preference of a renderer.

[0053] According to a sixteenth aspect, there is provided a computer-readable medium comprising program instructions to cause an apparatus to at least obtain information regarding a relationship between the generated at least one spatial audio parameter, at least one of the two or more acquired audio scene-based sources, and at least one audio signal associated with the generated at least one audio source; obtain a user position value and a user orientation value; request selection of the generated at least one audio source and / or at least two of the two or more audio scene-based sources based on the user position value; obtain at least one rendering source spatial parameter based on the request; obtain at least one rendering source audio signal based on the request; and generate at least one output audio signal based on the user orientation value, the at least one rendering source spatial parameter, and the at least one rendering source audio signal.

[0054] The electronic device may comprise a device as described herein.

[0055] The chipset may include an apparatus as described herein. [Brief description of the drawings]

[0056] For a better understanding of the present application, reference will now be made, by way of example, to the accompanying drawings, in which: [Figure 1] FIG. 1 illustrates diagrammatically a scenario audio scene in which the embodiment may be implemented. [Diagram 2] FIG. 2 is a flow diagram of an example of the current operations employed in rendering six degree of freedom higher order Ambisonics. [Diagram 3] FIG. 3 illustrates a schematic diagram of the scenario audio scene shown in FIG. 1 with the addition of a position-interpolated high-order Ambisonics source, in which an embodiment may be implemented. [Figure 4]FIG. 4 illustrates a schematic data structure for higher order Ambisonic sources and position-interpolated higher order Ambisonic sources as illustrated in FIG. 3 according to some embodiments. [Diagram 5] FIG. 5 illustrates an exemplary device in which some embodiments may be employed. [Figure 6] FIG. 6 is a flow diagram of the operation of the example apparatus shown in FIG. 5 according to some embodiments. [Figure 7] FIG. 7 is a flow diagram of renderer HOA source selection criteria for MPHOA rendering with rendering metadata including interpolated higher order Ambisonic and higher order Ambisonic sources. [Figure 8] FIG. 8 illustrates a system in which embodiments may be implemented. [Figure 9] FIG. 9 is a schematic diagram of an exemplary apparatus suitable for implementing the illustrated apparatus. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0057] Multi-point Higher Order Ambisonics (MPHOA) rendering is typically computationally intensive: the rendering process requires audio signals from multiple Higher Order Ambisonics (HOA) audio sources, which typically requires a large bandwidth to transmit audio signals from multiple audio sources.

[0058] Existing systems typically require the delivery of audio signals for three HOA sources and spatial metadata sets for three to five HOA sources (e.g., if the listener position is on the boundary of two triangles formed by five HOA sources).

[0059] Therefore, currently higher order Ambisonics-based systems require large computational power and large bandwidths.

[0060] For example, as shown in Fig. 1, an example of a scenario in which embodiments may be applied and provide advantages over current approaches is shown. In Fig. 1, four audio sources are shown in an audio scene, a first audio source AS1101, a second audio source AS2103, a third audio source AS3105, and a fourth audio source AS4107, which can be captured by six microphones (or microphone arrays) to create captured Higher Order Ambisonics (HOA) sources, a first HOA source H1111, a second HOA source H2113, a third HOA source H3115, a fourth HOA source H4117, a fifth HOA source H5119, and a sixth HOA source H6121. In the following example, a first subset S1123 of HOA sources may be defined that includes HOA sources H2113, H3115, and H4117, and a second subset S2125 of HOA sources may be defined that includes HOA sources H3115, H4117, and H5119. Depending on the listener position following the path represented by the curved line connecting listening positions P1131 and P2133, the renderer will request spatial metadata and audio signal data for three to five HOA sources. Although the examples have described captured HOA sources, in some embodiments there may also be synthetic HOA sources.

[0061] FIG. 2 shows the current method employed to render a 6DoF HOA system.

[0062] 2, Encoder Input Format (EIF) information is received or otherwise obtained, as shown in step 201. EIF (Encoder Input Format) is a form of content creator specified audio scene description. HOA sources H1, H2, H3, H4, H5 are included in the EIF (or an equivalent content creator scene description).

[0063] Further, in FIG. 2, audio signal data in MPEG-H or other format is received or otherwise obtained, as shown at step 202 .

[0064] In FIG. 2, the EIF and audio signal data is sent to an MPEG-I MPHOA encoder, as shown at step 203 .

[0065] Next, in FIG. 2, as shown at step 205, the encoder is configured to analyze the EIF to determine the number of HOA groups to process together to perform MPHOA processing to enable listener operation with six degrees of freedom.

[0066] Next, in FIG. 2, as shown at step 207, the encoder may be configured to determine higher order Ambisonic sources (OH) in the HOA group.

[0067] The encoder is then configured to process each of the HOA sources to generate the spatial metadata required for 6DOF rendering. The generation of spatial metadata for the higher order Ambisonic sources from the higher order Ambisonic audio signal is shown in FIG. 2 at step 209.

[0068] These operations may then be organized into an operation to generate rendering metadata based on the EIF, as shown in FIG. 2 at 200 .

[0069] The HOA source in the EIF is called the original HOA source because it is what the content creator defined in the EIF.

[0070] In FIG. 2, as shown at step 211, the (playback device) player is configured to select HOA source spatial metadata and audio signals based on the listener position (LP).

[0071] With reference to the scenario shown in Figure 1, if the listener is at position P1 131, the selection of sources is subset S1 123, and for position P2 133 the selection is subset S2 125. The selection of OH source spatial metadata and audio signals based on the listener position (LP) is shown in Figure 2 as step 211.

[0072] The selected content is then retrieved. This operation typically consumes a significant amount of bandwidth. The operation of retrieving the OH source spatial metadata and audio signals that form a triangle around the LP is shown in FIG. 2 as step 213.

[0073] In FIG. 2, as shown in step 210, steps 211 and 213 can be summarized as content selection and retrieval based on LP (3-5 OH source spatial metadata and audio signals).

[0074] The retrieved subset is then input for rendering processing in a playback or rendering device. The execution of LP-based HOA spatial metadata interpolation starting from the nearest HOA audio signal is shown in FIG. 2 as step 221. This rendering process is a computationally intensive operation since it involves data from multiple HOA sources. Step 221 can be summarized as the computational resources required for processing and memory for spatial metadata interpolation from 3-5 OH sources.

[0075] In the following description, an original Higher Order Ambisonics source (OH source) is an HOA or scene-based audio provided by a content creator as part of the creation of an audio scene. An original HOA (OH) source is generated from either a microphone (or microphone array) capturing the scene from one or more positions in the audio scene. HOA sources can also be synthetically generated from a set of audio objects. OH sources are introduced into the scene at content creation time, and therefore are present in the audio scene description. In the MPEG-I immersive audio standard, the content creator's audio scene description is EIF (Encoder Input Format).

[0076] Further in the following description, a positionally interpolated HOA source (PIH source) is an HOA source that is generated as part of the rendering metadata creation to allow for greater flexibility in content consumption selection. Since the PIH source is introduced at the 6-DOF rendering metadata creation stage by the MPEG-I encoder, the MPEG-I immersive audio standard does not have PIH sources present in the content creator's scene description or EIF. However, the PIH source is present in the MPEG-I bitstream that is available to the MPEG-I player for content selection and retrieval.

[0077] The concepts illustrated in the following embodiments describe a method and apparatus for reducing the rendering calculations and network bandwidth required for 6DOF rendering for scenes containing multiple original HOA (OH) sources by generating additional positionally interpolated HOA (PIH) sources (with spatial metadata) during the rendering metadata creation stage and generating associated HOA source audio information for the PIH sources to enable 6DoF translation rendering with only a single HOA source.

[0078] The pre-generated PIH sources and associated metadata can be hosted by a content delivery server along with the OH sources, so that renderers and players can directly retrieve the appropriate PIH sources and the indicated audio signals. This results in less computation, including single HOA source metadata processing (instead of the typically three HOA source metadata), and reduced bandwidth due to the need to retrieve only the single HOA source spatial metadata and associated OH source audio signals.

[0079] Thus, low complexity, low bandwidth mobile playback can be achieved in some embodiments by: generating one or more PIH sources including spatial metadata interpolated from the OH audio signal during rendering metadata creation; determining at least one HOA audio signal associated with each of the PIH sources based on at least one criterion (e.g., closest to a location of the PIH source, second closest but closer to another adjacent PIH source); generating information relating between the PIH spatial metadata and the HOA audio signal; generating HOA source information including OH and PIH representations to a renderer; This can be achieved by: In some embodiments, the PIH source and OH source metadata are indicated in a media presentation description (or manifest) to enable content selection for Dynamic Adaptive Streaming over HTTP (DASH)-based delivery. In some embodiments, the renderer may be configured to operate in different modes depending on bandwidth and / or computational resource constraints. In some embodiments, there may be three modes: Mode 1: Rendering with spatial metadata and audio signals of two or more OH sources. Mode 2: Rendering with a single OH source spatial metadata and associated OH audio signal with limited 6DoF. Mode 3: Rendering with the closest single OH or PIH source spatial metadata and associated OH audio signal for 3DoF rendering.

[0080] Mode 1 allows more freedom of user movement. Mode 2 allows less freedom of user movement compared to Mode 1, but requires less computational effort and less bandwidth compared to Mode 1. Mode 3 is the least computational effort compared to Modes 1 and 2, but is expected to have OH and PIH sources available at the expected listening positions. If the hotspots of the listener positions are known, the MPEG-I encoder can generate PIH sources at the appropriate LPs. In one embodiment of the present invention, the content processing server (i.e., hosting the MPEG-I encoder) periodically updates the available PIH sources based on the crowd-sourced content consumption hotspots obtained by the LP trajectory collected from the user movements. In any of the above modes, the player can perform rendering in a comparable manner with PIH sources in all three modes. The need for additional PIH sources is due to limited user movement without significant loss of sound quality with a single PIH rendering (limited 6DoF) compared to using a subset of OH sources for full 6DoF.

[0081] By employing the embodiments described herein, the MPHOA rendering bandwidth can be reduced by up to 1 / 3. Furthermore, MPHOA rendering on compute-constrained consumer devices can make low-end equipment a target market for 6DoF MPHOA rendering.

[0082] In the embodiments described herein, a trade may be made between computational complexity at the renderer / player and network bandwidth requirements by utilizing additional storage and pre-computation at the server.

[0083] This allows for flexible device configurations when rendering complex scenes.

[0084] In an embodiment, generating positionally interpolated HOA or positionally interpolated HOA (PIH) sources with the help of OH sources and adding appropriate OH audio source information at the 6DoF rendering metadata creation stage is a concept not currently discussed elsewhere. As indicated above, this shifts the computational complexity from the renderer to content processing or content hosting (e.g., a DASH server hosting the PIH source spatial metadata) without impacting content creation.

[0085] Additionally, the concepts described herein are adapted to present a single OH audio source signal which significantly reduces the need for distribution of multiple HOA source audio signal data.

[0086] Although the concepts and embodiments are suitable for implementation within the MPEG standard, the techniques can be applied to other formats and spatial audio capture content. Furthermore, the embodiments described herein are configured to illustrate the use of only the best fit, without requiring modification of the audio signal data.

[0087] The inventive concept will now be described with reference to Fig. 3, which shows a scene as shown in Fig. 1. However, in this example scene there is also an additional fifth audio source AS5109 and additional positions for listener devices 135 and 137. The scene further includes (position corrected HOA) PIH sources 301, 303, 305 and 307. The PIH sources create additional listening positions at which the renderer can perform rendering by obtaining only a single HOA source spatial metadata and a single HOA source audio signal data.

[0088] With reference to FIG. 4, an example of the data structure of a PIH source 410 and HOA sources 420, 430, 440 associated with a scene label 401 "poal" is shown.

[0089] In this example, the data structure of OH (HOA) sources such as sources 420, 430, 440 is as follows: Source type identifier (hoa_source_type OH) 441 Unique source identifier (hoa_source_id) 443 HOA Source Information 445 6DOF HOA Audio Data Track Header 447 HOA Audio Signal Datatrack 449

[0090] The data structure of a PIH source, such as source 410 in this embodiment, is as follows: Source type identifier (hoa_source_type PIH) 411 Unique source identifier (hoa_source_id) 413 HOA Source Information 415 6DOF HOA Spatial Metadata Track Header 417 HOA Rendering Metadata Track 419

[0091] With reference to FIG. 5, an exemplary system of devices suitable for implementing some embodiments is shown.

[0092] In this example, the system includes an EIF 502 input that is configured to be passed to an encoder (in this example, an MPEG-I encoder 505).

[0093] In addition, the system includes an MPEG-I audio 504 input configured to be passed to an encoder 505 .

[0094] The system further comprises an (MPEG-I) encoder 505. The encoder 505 is configured to receive the (MPEG-I) audio 504 and the EIF 502 and generate rendering metadata from the received scene description (EIF) and raw audio signal.

[0095] In some embodiments, the encoder uses scene description information (EIF) to detect or determine the presence of one or more HOA groups. Each HOA group includes two or more HOA sources. The HOA sources specified by the content creator in the EIF information are called original HOA sources or OH sources.

[0096] Additionally, the encoder 505 is configured to determine at least one candidate position for generating an additional position-interpolated HOA source (PIH source).

[0097] The encoder 505 in some embodiments performs spatial metadata interpolation using the OH sources that encompass each of the candidate PIH sources to generate position-interpolated spatial metadata. Methods for performing spatial metadata interpolation can be as discussed in GB application 2002710.8.

[0098] In some embodiments, the encoder is further configured to determine an audio signal from one or more OH sources used to calculate the PIH spatial metadata used.

[0099] For example, in some embodiments, the closest OH source audio signal is added as the OH source audio signal associated with a particular PIH source.

[0100] In some embodiments, the encoder is configured to select the OH source audio signal such that it is an audio signal that is also associated with a nearby PIH source. Such an approach allows the player / renderer to obtain a longer duration of audio content to ensure seamless operation in response to listener movements.

[0101] In some embodiments, the number of identified or determined listener positions may depend on the number of OH sources and the distance between the OH sources.

[0102] Moreover, in some embodiments, the determined number of PIH sources depends on the amount of movement allowed. The number of PIH sources depends on the tradeoff between the range of movement allowed for each PIH source and the allowable size of storage.

[0103] For example, in simple terms, the microphones are arranged (for each microphone in the audio scene) to form an equilateral triangle, with the area A of the triangle being given by

number

[0104] If the PIH sources are embedded among the OH sources such that the travel distance of each PIH source is one sixth, then the region of interest for 6DoF rendering by each PIH source is (a / d)^2*π, where d>1. Typically, parametric rendering of a single PIH source can maintain high quality for limited travel distances, although there is noticeable degradation as d approaches 1.

[0105] The number of PIH sources covering A is given by:

number

[0106] If a=3m, then A=3.9 square meters. For a PIH with 0.5m of movement (i.e., d=6), the number of PIH sources required is 5 in this example.

[0107] The definition for an equilateral triangle can be extended to any triangle.

[0108] In some embodiments, the inter-PIH distance may be determined such that the additional storage capacity required is a constraint.

[0109] Furthermore, in some embodiments, only a subset of the OH sources are embedded with the PIH sources. For example, this subset implementation can be employed in large audio scenes where additional data storage with the PIH sources is controlled based on a listener position heatmap.

[0110] Additionally, the selection of this subset can be customized per region based on the rendering metadata and the CDN (content delivery network) hosting the audio signal data.

[0111] In some embodiments, the HOA source information for the OH source can be as follows: aligned(8) HOASourceInformationStruct(){ unsigned int(2) hoa_source_type; / / OH or PIH source HOASourcePositionStruct(); / / position of the HOA source unsigned int(16) hoa_source_id; / / unique identifier for each HOA source unsigned int(3) hoa_order; / / order of HOA source bit (3) reserved = 0; HOAGroupInformationStruct(); / / grouping information of the HOA source } aligned(8) HOAGroupInformationStruct(){ unsigned int(16) hoa_source_group_id; / / Unique HOA group identifier }

[0112] [Table 1]

[0113] As shown in FIG. 4, grouping of OH sources, PIH sources, and OH audio signals that are authoritative alternatives for rendering 401 can be defined using an EntityToGroupBox with grouping_type equal to “poal” (PIH and OH source audio alternative), which specifies that tracks containing PIH source metadata and associated alternative OH audio signals are included in the same entity group. aligned(8) class HOASourceOHAudioAlternativesBox(version, flags) extends EntityToGroupBox('poal', version, flags) { / / conditionally mandatory for(i=0; i <num_entities_in_group; i++) unsigned int(16) ref_ohaudio_id[i]; }

[0114] In this example, ref_ohaudio_id[i] specifies the hoa_source_id from the track identified by the i-th entity_id that is the audio signal that is reliable for rendering the PIH sources in this group. The entity_ids of OH audio signals suitable for rendering the PIH sources are ordered such that the smallest index has the highest priority. The OH identified by ref_ohaudio_id[0] is the most preferred OH audio signal source. The i-th referenced track may have a hoa_source_id equal to ref_ohaudio_id[i]. If a single audio signal is suitable, the number of entities may be absent.

[0115] In some implementations, a 6DOF OH source is indicated by signaling a HOASourceInformationStruct() as a new box -6DOFOHSourceBox ("6dohb") included in the sample entry of the spatial rendering metadata track, carrying information about the associated OH audio signal data. aligned(8) 6DOFHOABox() extends FullBox('6dhb',0,flags) { #container: AudioSampleEntry or Timed metadata. New definition HOASourceInformationStruct(); unsigned int(1) hoa_source_audio_or_render_meta; bit(7) reserved = 0; }

[0116] In some embodiments, the (MPEG-I) encoder 505 is further configured to render or generate a metadata manifest for the content selection.

[0117] In some embodiments, the generation of a metadata manifest for content selection is such that for a DASH Media Presentation Description (MPD), an HOA source element with an @schemeIdUri attribute equal to "urn:mpeg:mpegI:mia:2021:6DOH" is referred to as the original HOA source (an OH source defined in EIF), a 6DOH descriptor. Further, the HOA source is described in a HOASourceInformationStruct(), where the value of hoa_source_type is equal to 0.

[0118] Also, an HOA source element with an @schemeIdUri attribute of "urn:mpeg:mpegI:mia:2021:6DPH" is referred to as a positional interpolation HOA source (PIH source), 6DPH descriptor. HOA sources are described in HOASourceInformationStruct(), and the value of hoa_source_type is equal to 1.

[0119] In some embodiments, a number of 6DOH adaptation sets exist for the data (audio signal data and spatial metadata) for each of the OH sources in the content creator scene description. Similarly, if there are one or more PIH sources added in the rendering metadata creation stage, they exist as adaptation sets with a corresponding 6DPH descriptor for each of the interpolated spatial metadata representations.

[0120] In the absence of a PIH source, rendering is performed using only the OH sources (audio and spatial metadata). In the presence of a PIH source in the media manifest, the player is free to choose an appropriate adaptation set for retrieval and playback depending on the computational resources and bandwidth availability of the rendering device.

[0121] The 6DOH and 6DPH descriptors in some embodiments shall include a HOASourceInfo element with an @value attribute and the sub-elements and attributes defined in the table below.

[0122] [Table 2-1] [Table 2-2]

[0123] In some embodiments, another manifest implementation approach includes a single descriptor for all HOA sources (OH and PIH). The Media Presentation Description (MPD) in such embodiments has an additional required parameter hoa_source_type that indicates whether the adaptation set represents an OH source or a PIH source.

[0124] In some embodiments, the OH and PIH sources are listed as attributes in Java Script Object Notation (JSON) format. This may be useful when a delivery method other than DASH is used. Depending on the delivery method preference, the Session Description Protocol (SDP) may also be used to describe the available HOA sources. This is beneficial for broadcast and multicast delivery of content. In such a scenario, the player can select an approach stream representing an OH or PIH source to perform 6DOF rendering.

[0125] After the rendering bitstream and HOA source audio signal 506 are generated, they can be passed to an appropriate MPEG-I content node 508 (which may be a server or a cloud-based storage element).

[0126] The content node 508 can further forward one positionally interpolated HOA source metadata and one HOA source audio 510 to an (MPEG-I) renderer 511.

[0127] In some embodiments, the renderer 511 may be configured to operate in different modes to take advantage of the presence of OH and PIH sources in the 6DoF audio scene. In some embodiments, the modes may be selected based on the determined estimated computational resource requirements and available network bandwidth, since different players have preferences depending on the availability of computational resources and network bandwidth.

[0128] In the examples below, the following modes are presented, although other modes of operation may also be implemented:

[0129] In some embodiments, there is a first rendering mode (Mode1), which may be adopted in some embodiments where the renderer 511 is computationally equipped to perform state-of-the-art MPHOA rendering. In this mode of operation, the renderer 511 is configured to obtain OH sources and corresponding audio signals that form a triangle based on the listener position.

[0130] This mode has the advantage that the listener has more freedom of movement since (usually) more than two OH sources can be used, and also has the advantage that more data can be acquired in advance since the listener is likely to move within a triangle that contains three OH sources.

[0131] In this mode, in addition to the audio signal data of the OH source, the renderer can also use spatial metadata generated specifically for the OH source.

[0132] Additionally, this mode requires bandwidth to retrieve not only the audio signals but also the spatial metadata for 3-5 HOA sources, since the renderer may switch between one of the audio signals depending on the listener's position.

[0133] However, with the help of the proposed OH source audio signal data, in some embodiments, any suitable MPHOA rendering may be optimized by requiring the use of only the "optimal" OH source audio signal data, where "optimal" may be either the closest or the least variable, depending on the implementation.

[0134] In some embodiments, there is a second rendering mode (Mode2). This mode can be implemented when the renderer 511 has computational power but is constrained by bandwidth. In such an embodiment, the renderer 511 can be configured to obtain based on the listener position, one PIH source spatial metadata, and the associated OH source audio signal. The renderer can perform limited six degree of freedom movements with the retrieved data. The renderer 511 can be configured in some embodiments to obtain the next adjacent PIH source spatial metadata and associated source audio signal for rendering.

[0135] Therefore, in this mode, it is only necessary to acquire spatial metadata for a single PIH source and audio signal data for a single OH source.

[0136] Furthermore, the second rendering mode is configured to generate spatial metadata for PIH sources in addition to OH sources, and therefore requires additional storage on the content node 508 (e.g., a CDN for DASH delivery).

[0137] In an embodiment, this mode may be employed when the renderer 511 is computationally and bandwidth constrained. In such an embodiment, the rendering mode is configured such that the renderer 511 obtains PIH source spatial metadata and associated audio signals. The renderer 511 in such an embodiment is configured to perform limited 6DOF rendering.

[0138] In some embodiments, there is a third rendering mode (Mode3), in which the renderer 511 is significantly computationally constrained. In such an embodiment, the renderer 511 is configured to select the closest PIH source metadata and associated OH audio signal data to perform 3DOF rendering only.

[0139] In this third rendering mode, the renderer 511 is configured to provide a better listening experience compared to using the nearest OH source (which may be the default behavior) when the renderer is only able to perform 3DOF rendering due to computational constraints. A further advantage is that content creators do not need to do additional content creation to provide a spatially localized experience for computationally constrained renderers. The MPEG-I encoder generates the necessary PIH sources when creating spatial metadata for the rendering.

[0140] With reference to FIG. 6, the operation of the system and the operation of the renderer for modes 1 and 2 are shown.

[0141] Thus, in FIG. 6, step 601 illustrates the act of receiving an EIF.

[0142] Further illustrated in FIG. 6 is an operation for receiving MPEG-H audio, as shown at step 602 .

[0143] Then, as shown in step 603 in FIG. 6, when EIF information and MPEG-H audio are received, the operation proceeds to encoding MPEG-I MPHOA.

[0144] In FIG. 6, after generating the MPEG-I MPHOA, the MPHOA groups are determined, as shown in step 605 .

[0145] Further shown in FIG. 6 is the determination of the OH source for the HOA group, as shown at step 607 .

[0146] Steps 609, 611, 613, and 615 describe a method based on an OH source and not using a PIH source.

[0147] For example, in FIG. 6, as shown at step 609, a first mode of rendering illustrates the generation of OH spatial metadata from an OH audio signal.

[0148] Further, in FIG. 6, as shown at step 611, a first mode of rendering illustrates the selection of OH source spatial metadata and audio signals based on the listener position (LP).

[0149] Further, in FIG. 6, as indicated at step 613, a first mode of rendering illustrates the acquisition of OH source spatial metadata and audio signals that form a triangle around the LP.

[0150] Next, in FIG. 6, as shown at step 615, a first mode of rendering is shown to render the HOA spatial metadata interpolation based on the LP starting from the closest HOA audio signal.

[0151] Steps 610, 612, 614, 616 describe a method based on additional encoder generated PIH sources and associated OH audio data signaling.

[0152] For example, in FIG. 6, as shown at step 610, in the second rendering mode, an operation of determining an interpolation position for generating a PIH source is shown.

[0153] Further, in FIG. 6, a second mode of rendering is shown as generating a PIH source that includes spatial metadata and associated audio signal information, as indicated at step 612 .

[0154] Further, in FIG. 6, a second mode of rendering is shown to obtain a single PIH source spatial metadata and associated audio signal based on the LP, as indicated at step 614 .

[0155] Next, in FIG. 6, as shown at step 616, a second mode of rendering is shown to be rendering from the PIH source spatial metadata and associated audio signals based on the LP, starting from the closest HOA audio signal.

[0156] FIG. 7 shows a flow diagram of the renderer HOA source selection criteria for MPHOA rendering with rendering metadata including interpolated higher order Ambisonics and higher order Ambisonic sources.

[0157] Thus, in FIG. 7, as indicated at step 701, some embodiments begin playing an MPHOA scene that includes OH and PIH sources.

[0158] Then, in FIG. 7, a determination is made as to whether there are sufficient computational resources in the rendering device and acquisition bandwidth, as shown at step 703 .

[0159] 7, if there are sufficient computational resources on the rendering device and playback bandwidth, playback begins with OH spatial metadata and audio signals from sources forming triangles based on LP, as shown in step 707. In other words, the first rendering mode or default mode is adopted.

[0160] In FIG. 7, as shown at step 705, if the rendering device and acquisition bandwidth do not have sufficient computational resources, a further check may be performed to determine whether there are sufficient computational resources to perform 3DoF+ rendering.

[0161] 7, if there are not enough computational resources, the playback starts 3DoF playback with the nearest PIH spatial metadata and audio signal based on LP, as shown in step 712. In other words, the third rendering mode (for low bandwidth and minimum complexity) is adopted.

[0162] 7, if there are sufficient computational resources to perform 3Dof+ rendering, playback begins with PIH spatial metadata and audio signals based on LP, as shown in step 711. In other words, the second rendering mode (for low bandwidth and low complexity) is adopted.

[0163] With reference to FIG. 8, a further diagram of a system is shown in which there is a content creator scope 800 configured to generate N OH sources 890 .

[0164] This section includes an audio input 803 and an EIF input or generator 801 .

[0165] Furthermore, the MPEG-H Encoder and Decoder 805 is configured to receive audio signals from an audio input and pass these to an MPEG-H Encoded / Decoded Audio Buffer / Store 807 .

[0166] The MPEG-H encoded / decoded audio buffer / storage 807 may be further configured to pass the encoded audio signal to an (MPEG-I) encoder 809 .

[0167] This section may further include an Encoder 809 (although this may be implemented within the Rendering Metadata Creation 820 unit), which is configured to obtain or receive EIF information, the (raw) audio signal from the Audio Input 803, and the encoded (MPEG-H) audio, and to generate a PIH source.

[0168] In some embodiments, there may be a rendering metadata creation 820 unit, which may include the encoder 809 or may obtain the output of the encoder 809, as indicated above.

[0169] The rendering metadata creation 820 may, in some embodiments, include a metadata renderer 821 configured to generate metadata, as described above.

[0170] Thus, the output of the rendering metadata creation 820 component is that there are (N) OH sources and further (M) PIH sources 892 .

[0171] A further section in the system is the content hosting for distribution 840. The content hosting for distribution 840 can provide an indication 894 of OH and PIH sources and the association of OH source audio with PIH sources.

[0172] The content hosting for distribution 840 in some embodiments includes an MPEG-I 6DoF content bitstream buffer / storage 841. The MPEG-I 6DoF content bitstream buffer / storage 841 is configured to receive or obtain the OH and PIH sources in the bitstream and provide appropriate buffer / storage elements to hold them.

[0173] The content hosting for distribution 840 further includes a content manifest selector 843. The content manifest selector 843 is configured to generate and output to the playback device 861 a manifest 862, and spatial metadata and audio data 864.

[0174] The playback 860 in some embodiments is configured to implement various rendering modes 896, such as OH source-based rendering and PIH source-based rendering.

[0175] In some embodiments, the playback device 861 comprises a player 863. The player 863 further comprises an MPHOA renderer 865 and a content selector 867. The player 863 is configured to output the renderer audio as a headphone output 866 to a headphone / tracker, and further configured to obtain 6DoF tracking information 868 therefrom.

[0176] 9, an example of an electronic device that can be used as a computer, an encoder processor, a decoder processor, or any of the functional blocks described herein is shown. The device can be any suitable electronic device or device. For example, in some embodiments, the device 1600 is a mobile terminal, a user equipment, a tablet computer, a computer, an audio playback device, etc.

[0177] In some embodiments, the device 1600 includes at least one processor or central processing unit 1607. The processor 1607 may be configured to execute various program code, such as the methods as described herein.

[0178] In some embodiments, the device 1600 comprises a memory 1611. In some embodiments, at least one processor 1607 is connected to the memory 1611. The memory 1611 may be any suitable storage means. In some embodiments, the memory 1611 comprises a program code section for storing program code implementable in the processor 1607. Furthermore, in some embodiments, the memory 1611 may further comprise a storage data section for storing data, e.g. data that has been processed or is to be processed according to the embodiments as described herein. The implementation program code stored in the program code section and the data stored in the storage data section may be retrieved by the processor 1607 whenever needed via the memory-processor connection.

[0179] In some embodiments, the device 1600 has a user interface 1605. The user interface 1605, in some embodiments, may be coupled to a processor 1607. In some embodiments, the processor 1607 may control the operation of the user interface 1605 and receive input from the user interface 1605. In some embodiments, the user interface 1605 may allow a user to input commands to the device 1600, for example, via a keypad. In some embodiments, the user interface 1605 may allow a user to obtain information from the device 1600. For example, the user interface 1605 may include a display configured to display information from the device 1600 to the user. The user interface 1605, in some embodiments, may have a touch screen or touch interface that can both allow information to be input into the device 1600 and also display information to a user of the device 1600.

[0180] In some embodiments, the apparatus 1600 has an input / output port 1609. The input / output port 1609 in some embodiments comprises a transceiver. The transceiver in such embodiments may be coupled to the processor 1607 and configured to enable communication with other apparatuses or electronic devices, for example, via a wireless communication network. The transceiver or any suitable transceiver or transmitter and / or receiver means may in some embodiments be configured to communicate with other electronic devices or apparatuses via a wired or non-wired connection.

[0181] The transceiver may communicate with the further device by any suitable known communication protocol, for example in some embodiments the transceiver may use a suitable Universal Mobile Telecommunications System (UMTS) protocol, a Wireless Local Area Network (WLAN) protocol such as IEEE 802.X, a suitable short range radio frequency communication protocol such as Bluetooth, or an Infrared Data Path (IRDA).

[0182] The transceiver input / output port 1609 can be configured to transmit / receive audio signals, bitstreams, by using a processor 1607 executing appropriate code, and in some embodiments, to perform the operations and methods as described above.

[0183] In general, various embodiments of the present invention may be implemented in hardware or special purpose circuits, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware, and other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the present invention is not limited thereto. Although various aspects of the present invention may be illustrated and described as block diagrams, flow charts, or using some other graphical representation, it will be appreciated that the blocks, apparatus, systems, techniques, or methods described herein may be implemented in, by way of non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware, or controller, or other computing device, or any combination thereof.

[0184] The embodiments of the present invention may be implemented by computer software executable by a data processor of a mobile terminal such as a processor entity, or by hardware, or by a combination of software and hardware. Furthermore, in this regard, it should be noted that any blocks of the logic flow as illustrated may represent program steps or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media such as memory chips or memory blocks implemented within a processor, magnetic media, and optical media.

[0185] The memory may be of any type suitable for the local technology environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed and removable memories, etc. The data processor may be of any type suitable for the local technology environment and may include, by way of non-limiting examples, one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), gate level circuits, and processors based on multi-core processor architectures.

[0186] Embodiments of the present invention can be implemented in a variety of components, such as integrated circuit modules. The design of integrated circuits is generally a highly automated process. Complex and powerful software tools are available for converting logic level designs into semiconductor circuit designs suitable for etching onto semiconductor substrates.

[0187] Programs such as those from Synopsys, Inc. of Mountain View, Calif., and Cadence Design, Inc. of San Jose, Calif., use established design rules and libraries of prestored design modules to automatically route conductors and place components on a semiconductor chip. Once the semiconductor circuit design is complete, the design is sent in a standardized electronic format (Opus, GDSII, etc.) to a semiconductor manufacturing facility (fab) for manufacture.

[0188] The above description has provided a full and informative description of the exemplary embodiments of the present invention, by way of illustrative and non-limiting examples. However, various changes and modifications will become apparent to those skilled in the art in light of the above description, when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications of the teachings of this invention will still fall within the scope of the present invention, as defined in the appended claims.

Claims

1. An apparatus comprising at least one processor and at least one memory containing computer program code, the at least one memory and the computer program code being transmitted by the at least one processor to the apparatus, the apparatus comprising at least: - acquiring two or more audio scene based sources, the two or more audio scene based sources being associated with one or more positions in an audio scene, each audio scene based source comprising at least one spatial parameter and at least one audio signal; determining at least one location associated with at least one of the two or more captured audio scene-based sources, the at least one location being determined for rendering; and generating at least one audio source based on the determined at least one position, the means for generating the at least one audio source being configured to: generating at least one spatial audio parameter based on the at least one spatial parameter associated with at least one of the acquired two or more audio scene-based sources in association with the determined at least one location; configured to generate at least one audio source signal for the at least one audio source based on the at least one audio signal of the acquired two or more audio scene-based sources in association with the determined at least one position; generating information regarding a relationship between the generated at least one spatial audio parameter and the at least one audio signal associated with at least one of the captured two or more audio scene-based sources, the generated at least one audio source being selected based on a renderer preference; An apparatus configured to cause

2. The device is adapted to determine the at least one location, the device is adapted to obtain the at least one location from the at least one further device, the device further comprising: transmitting the information to the at least one further device; upon selecting the two or more audio scene-based sources, outputting at least one spatial parameter and the at least one audio signal of the selected two or more sources; upon selecting the at least one audio source, outputting the at least one spatial audio parameter of the audio source and the at least one audio source signal; The apparatus of claim 1 configured to:

3. The device adapted to select the two or more audio scene based sources or the at least one audio source based on the position of the one from at least one further device, the bandwidth of a transmission or storage channel between said device and said further device, and the computing power of said further device, The apparatus of claim 2 , adapted to select the two or more audio scene-based sources or the at least one audio source based on at least one of:

4. 3. The apparatus of claim 2, wherein the apparatus adapted to generate the at least one audio source based on the determined at least one position is adapted to determine a position of the at least one audio source based on the determined at least one position from the at least one further device.

5. The device adapted to generate the at least one audio source based on the determined at least one position, selecting or defining a group of audio scene based sources within the two or more audio scene based sources; generating the at least one spatial audio parameter based on a combination of the two or more audio scene based sources and at least one spatial parameter from the selected or defined group of audio scene based sources within the two or more audio scene based sources; generating the at least one audio source signal based on a combination of the two or more audio scene based sources and at least one audio signal from the selected or defined group of audio scene based sources within the two or more audio scene based sources; 2. The apparatus of claim 1, adapted to:

6. The apparatus adapted to acquire the two or more audio scene-based sources comprises: acquiring at least two audio signals from microphones positioned in the audio scene; analyzing the at least two audio signals to identify the two or more audio scene-based sources and at least one spatial parameter and the at least one audio signal associated with each of the two or more audio scene-based sources; 2. The apparatus of claim 1, adapted to:

7. The apparatus of claim 1 , wherein the means configured to obtain two or more audio scene-based sources is configured to receive or synthesize the two or more audio scene-based sources.

8. The apparatus of claim 1 , wherein the two or more audio scene-based sources are high-order Ambisonics sources.

9. The apparatus of claim 1 , wherein the at least one audio source generated based on the determined at least one position is a positionally interpolated higher order Ambisonics source.

10. An apparatus comprising at least one processor and at least one memory containing computer program code, the at least one memory and the computer program code being transmitted by the at least one processor to the apparatus, the at least one memory being configured to: obtaining information regarding a relationship between the generated at least one spatial audio parameter and at least one audio signal associated with at least one of the two or more acquired audio scene-based sources and the generated at least one audio source; obtaining a user position value and a user orientation value; requesting a selection of the at least one generated audio source and / or at least two of the two or more audio scene-based sources based on the user position value; obtaining at least one rendering source spatial parameter based on the request; obtaining at least one rendering source audio signal based on the request; generating at least one output audio signal based on the user orientation value, the at least one rendering source spatial parameter, and the at least one rendering source audio signal; configured to cause The device adapted to request a selection of the at least one generated audio source and / or at least two of the two or more audio scene based sources based on the value of the user position further comprises: - the bandwidth of a transmission or storage channel between said device and a further device, from which said at least one rendering source spatial parameter and said at least one rendering source audio signal are obtained, and the computing power of said device; determining at least one of: selecting the at least one audio source or at least two of the two or more audio scene-based sources based on the bandwidth or the computational power of the transmission or storage channel; An apparatus adapted to perform the above.

11. 1. A method for an apparatus for generating an immersive audio scene, the method comprising: - acquiring two or more audio scene based sources, the two or more audio scene based sources being associated with one or more positions in an audio scene, each audio scene based source comprising at least one spatial parameter and at least one audio signal; determining at least one location associated with at least one of the two or more captured audio scene-based sources, the at least one location being determined for rendering; and generating at least one audio source based on the determined at least one location, the generating the at least one audio source comprising: generating at least one spatial audio parameter based on the at least one spatial parameter associated with at least one of the acquired two or more audio scene-based sources in association with the determined at least one location; generating at least one audio source signal for the at least one audio source based on the at least one audio signal of the acquired two or more audio scene-based sources in association with the determined at least one position; generating generating information regarding a relationship between the generated at least one spatial audio parameter and the at least one audio signal associated with at least one of the captured two or more audio scene-based sources, the generated at least one audio source being selected based on a renderer preference; The method includes:

12. Determining at least one location associated with at least one of the two or more captured audio scene-based sources includes obtaining the at least one location from at least one further device, the method further comprising: transmitting said information to said at least one further device; upon selecting the two or more audio scene-based sources, outputting at least one spatial parameter and the at least one audio signal of the selected two or more sources; upon selecting the at least one audio source, outputting the at least one spatial audio parameter of the audio source and the at least one audio source signal; The method of claim 11 , comprising:

13. Selecting the two or more audio scene-based sources or the at least one audio source based on a position from at least one further device includes: the bandwidth of a transmission or storage channel between said device and said further device, and the computing power of said further device, The method of claim 12 , further comprising selecting the two or more audio scene-based sources or the at least one audio source based on at least one of:

14. 13. The method of claim 12, wherein generating the at least one audio source based on the determined at least one location comprises determining the location of the at least one audio source based on the determined at least one location from the at least one further device.

15. Generating the at least one audio source based on the determined at least one location includes: selecting or defining a group of audio scene based sources within the two or more audio scene based sources; generating the at least one spatial audio parameter based on a combination of the two or more audio scene based sources and at least one spatial parameter from the selected or defined group of audio scene based sources within the two or more audio scene based sources; generating the at least one audio source signal based on a combination of the two or more audio scene based sources and at least one audio signal from the selected or defined group of audio scene based sources within the two or more audio scene based sources; The method of claim 11 , comprising at least one of:

16. Obtaining the two or more audio scene-based sources includes: acquiring at least two audio signals from microphones positioned in the audio scene; analyzing the at least two audio signals to identify the two or more audio scene-based sources and the at least one spatial parameter and the at least one audio signal associated with each of the two or more audio scene-based sources; The method of claim 11 , comprising at least one of:

17. The method of claim 11 , wherein obtaining the two or more audio scene-based sources comprises receiving or synthesizing the two or more audio scene-based sources.

18. A method for an apparatus for spatial audio signal rendering, the method comprising: obtaining information regarding a relationship between at least one generated spatial audio parameter and at least one audio signal associated with at least one of the two or more acquired audio scene based sources and the at least one generated audio source; obtaining a user position value and a user orientation value; requesting selection of the at least one generated audio source and / or at least two of the two or more audio scene based sources based on the user position value; obtaining at least one rendering source spatial parameter based on the request; obtaining at least one rendering source audio signal based on the request; and generating at least one output audio signal based on the user orientation value, the at least one rendering source spatial parameter, and the at least one rendering source audio signal; Requesting a selection of the at least one generated audio source and / or at least two of the two or more audio scene-based sources based on the value of the user position includes: - the bandwidth of a transmission or storage channel between said device and a further device, from which said at least one rendering source spatial parameter and said at least one rendering source audio signal are obtained, and the computing power of said device; determining at least one of: selecting the at least one audio source or at least two of the two or more audio scene-based sources based on the bandwidth or the computational power of the transmission or storage channel; A method comprising:

Citation Information

Patent Citations

  • Concept for generating extended or modified sound field descriptions using multi-point sound field descriptions

    JP2020527746A

  • Audio rendering with spatial metadata interpolation

    WO2021170900A1