Communication audio processing method and device in immersive audio scene rendering

Abstract

To address problems of the prior art.SOLUTION: A device for rendering a communication audio signal within an immersive audio scene includes means configured to be able to perform: obtaining at least one spatial audio signal for rendering within the immersive audio scene; obtaining the communication audio signal and positional information related to the communication audio signal; obtaining a rendering processing parameter related to the communication audio signal; determining a rendering method based on the rendering processing parameter; and selecting an insertion point in rendering processing of the determined rendering method, and / or a rendering element for the determined rendering method based on the rendering processing parameter.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] This application relates to a method and apparatus for processing communication audio within an augmented reality rendering, but is not limited to a method and apparatus for processing communication audio within an augmented reality six degree of freedom rendering. [Background technology]

[0002] Augmented reality (AR) applications (and other similar virtual scene creation applications such as mixed reality (MR) and virtual reality (VR)) that present virtual scenes to a user wearing a head mounted device (HMD) have become increasingly complex and advanced over time. An application may contain data that includes visual components (or overlays) and audio components (or overlays) that are presented to the user. These components are provided to the user according to the user's position and orientation (in the case of six degree of freedom applications) within the augmented reality (AR) scene.

[0003] Scene information for rendering an AR scene typically includes two parts. One part is virtual scene information that can be described during content creation (or by a suitable capture device or device) and represents the captured (or initially generated) scene. The virtual scene may be provided in Encoder Input Format (EIF) data format. The EIF and (captured or generated) audio data can be used by an encoder to generate a scene description and spatial audio metadata (and audio signals) that can be delivered via a bitstream to a rendering (playback) device or equipment. Thus, the scene description of an AR or VR scene is specified by the content creator during the content creation phase. In the case of VR, the entire scene is specified and rendered as specified in the content creator's bitstream.

[0004] The second part of the rendering of an AR audio scene is related to the physical listening space (or physical space) of the listener (or end user). The spatial information of the scene or listener can be obtained during the AR rendering (when the listener is consuming the content). This means that there is a fundamental aspect of AR that is different from VR, which means that the acoustic properties of the audio scene are only known during the content consumption (in the case of AR) and cannot be known or optimized during the content creation.

[0005] FIG. 1 shows an example of an AR scene where a virtual scene is located in a physical listening space. In this example, there is a user 107 located in the physical listening space 101. Furthermore, in this example, the user 109 is experiencing a six-degree-of-freedom (6DOF) virtual scene 113 with virtual scene elements. In this example, the elements of the virtual scene 113 are represented by two audio objects, a first object 103 (guitar player) and a second object 105 (drummer), a virtual occlusion element (e.g., represented as a virtual partition 117), and a virtual room 115 (e.g., a wall with a size, position, and acoustic material defined in the virtual scene description). A renderer (which in this example is a handheld electronic device or apparatus 111) is configured to perform the rendering such that the auralization is plausible with respect to the user's physical listening space (e.g., the wall positions and the acoustic material properties of the walls). The rendering is presented to the user 107 in this example by suitable headphones or headset 109.

[0006] Thus, for an AR scene, the content creator bitstream carries information about which audio and scene geometry elements correspond to which anchors in the listening space. As a result, the positions of audio elements, reflective elements, occlusion elements, etc. are only known during rendering. Furthermore, the acoustic modeling parameters are only known during rendering.

[0007] Social VR / AR is a further development of such systems, which are envisioned to support the rendering of voice and audio from other users in the virtual environment. Additionally, it has been proposed to render the received voice and audio communication as an immersive audio signal. Summary of the Invention [Problem to be solved by the invention]

[0008] SUMMARY OF THE DISCLOSURE The embodiments of the present application aim to address the problems of the prior art. [Means for solving the problem]

[0009] According to a first aspect, there is provided an apparatus comprising means configured to obtain at least one spatial audio signal for rendering within an immersive audio scene, obtain a communication audio signal and position information related to the communication audio signal, obtain rendering process parameters related to the communication audio signal, determine a rendering method based on the rendering process parameters, and determine an insertion point in the rendering process for the determined rendering method and / or a selection of rendering elements for the determined rendering method based on the rendering process parameters.

[0010] The means may be further configured to generate at least one output spatial audio signal from the at least one spatial audio signal and the communication audio signal based on the determined rendering method, an insertion point in the rendering process for the determined rendering method, and / or a selection of rendering elements for the determined rendering method based on the rendering process parameters.

[0011] The means may be further configured to determine at least one of an audio format, a tolerable delay value, and a communication audio signal delay associated with the communication audio signal.

[0012] The means configured to determine an insertion point in the rendering process for the determined rendering method and / or a selection of rendering elements for the determined rendering method based on the rendering process parameters may further be configured to at least one of: determine an insertion point in the rendering process based on at least one of an audio format, a permissible delay value, and a communication audio signal delay associated with the communication audio signal; and determine a rendering method and / or a selection of rendering elements for the determined rendering method based on at least one of an audio format, a permissible delay value, and a communication audio signal delay associated with the communication audio signal.

[0013] The tolerable delay value may be the amount of delay permitted for utilizing the communication audio signal, and the communication audio signal delay may be a delay value determined based on the end-to-end delivery delay and the delay in rendering the communication audio.

[0014] The audio format associated with the communication audio signal may include one of a one-way communication audio signal and a conversational communication audio signal between users in an immersive scene.

[0015] The means configured to generate at least one output spatial audio signal from the at least one spatial audio signal and the communication audio signal based on the determined rendering method and an insertion point in the rendering process for the determined rendering method and / or a selection of rendering elements for the determined rendering method based on the rendering process parameters may be configured to represent the communication audio signal as a high-order Ambisonic audio signal.

[0016] The means may be further configured to obtain user input, and the means configured to generate at least one output spatial audio signal from the at least one spatial audio signal and the communication audio signal based on the determined rendering method and an insertion point in the rendering process for the rendering method determined based on the rendering process parameters and / or a selection of rendering elements for the determined rendering method may be configured to generate the at least one output spatial audio signal from the at least one spatial audio signal and the communication audio signal further based on the user input, and the user input may be configured to define at least one of allowed communication audio signal types, allowed audio formats, allowed delay values, and at least one acoustic modeling preference parameter.

[0017] The means may be further configured to obtain a communication audio signal type associated with the at least one spatial audio signal, and the means configured to generate at least one output spatial audio signal from the at least one spatial audio signal and the communication audio signal based on the determined rendering method and an insertion point in the rendering process for the determined rendering method and / or a selection of rendering elements for the rendering method determined based on the rendering process parameters may be configured to generate the at least one output spatial audio signal from the at least one spatial audio signal and the communication audio signal further based on the at least one communication audio signal type associated with the at least one spatial audio signal.

[0018] The rendering processes and / or rendering elements may include one or more of Doppler processing, direct sound processing, material filtering processing, early reflection processing, diffuse late reverberation processing, sound source extension processing, occlusion processing, diffraction processing, sound source transformation processing, externalization rendering, and in-head rendering.

[0019] The means for determining an insertion point in a rendering process of the determined rendering method and / or a selection of rendering elements of the determined rendering method based on the rendering process parameters may be configured to determine a rendering mode, which may comprise a value indicative of an insertion point of the communication audio signal.

[0020] The value indicating the insertion point may include one of a first mode value indicating that the communication audio signal and the at least one spatial audio signal are inserted at the start of the rendering processing method, a second mode value indicating that the communication audio signal is to bypass the rendering processing and be directly mixed with the output of the rendering processing applied to the at least one spatial audio signal, and a third mode value indicating that the communication audio signal is to be partially rendered while the rendering processing is fully applied to the at least one spatial audio signal.

[0021] A third mode value indicating that the communication audio signal has been partially rendered may be a value indicating that the communication audio signal is a direct sound rendering of a point sound source and a binaural rendering relative to a user position.

[0022] The means may be further configured to determine an audio format type of the communication audio signal based on the rendering process parameters.

[0023] Furthermore, the means for determining an insertion point in a rendering process of the determined rendering method and / or a selection of rendering elements of the determined rendering method based on the rendering process parameters may be configured to determine an insertion point in a rendering process of the communication audio signal in the determined rendering method based on the audio format type.

[0024] The means configured to determine an insertion point in a rendering process for a communication audio signal in a rendering method determined based on an audio format type may be configured to determine that if the communication audio signal has an audio format type of a pre-rendered spatial audio format, the insertion point in the rendering method is in direct mixing with the output of the rendering process applied to the at least one spatial audio signal.

[0025] According to a second aspect, there is provided a method for an apparatus for rendering a communication audio signal within an immersive audio scene, the method comprising: obtaining at least one spatial audio signal for rendering within the immersive audio scene, obtaining the communication audio signal and position information related to the communication audio signal, obtaining rendering process parameters related to the communication audio signal, determining a rendering method based on the rendering process parameters, determining an insertion point in the rendering process for the determined rendering method, and / or selecting a rendering element for the determined rendering method based on the rendering process parameters.

[0026] The method may further include generating at least one output spatial audio signal from the at least one spatial audio signal and the communication audio signal based on the determined rendering method and an insertion point in the rendering process for the determined rendering method, and / or a selection of rendering elements for the determined rendering method based on the rendering process parameters.

[0027] The method may further include determining at least one of an audio format, a tolerable delay value, and a communication audio signal delay associated with the communication audio signal.

[0028] Determining an insertion point in the rendering process for the determined rendering method and / or selecting a rendering element for the determined rendering method based on the rendering process parameters may further include at least one of determining an insertion point in the rendering process further based on the determined at least one of an audio format, a permissible delay value, and a communication audio signal delay associated with the communication audio signal, and selecting a rendering method and / or a rendering element for the determined rendering method based on the determined at least one of an audio format, a permissible delay value, and a communication audio signal delay associated with the communication audio signal.

[0029] The tolerable delay value may be the amount of delay permitted for utilizing the communication audio signal, and the communication audio signal delay may be a delay value determined based on the end-to-end delivery delay and the delay in rendering the communication audio.

[0030] The audio format associated with the communication audio signal may include one of a one-way communication audio signal and a conversational communication audio signal between users in an immersive scene.

[0031] Generating at least one output spatial audio signal from the at least one spatial audio signal and the communication audio signal based on the determined rendering method and an insertion point in the rendering process for the determined rendering method, and / or a selection of rendering elements for the determined rendering method based on the rendering process parameters, may include representing the communication audio signal as a high-order Ambisonic audio signal.

[0032] The method may further include obtaining user input and generating at least one output spatial audio signal from the at least one spatial audio signal and the communication audio signal based on the determined rendering method and an insertion point in the rendering process for the determined rendering method and / or a selection of rendering elements for the determined rendering method based on the rendering process parameters, and may further comprise generating the at least one output spatial audio signal from the at least one spatial audio signal and the communication audio signal further based on the user input, where the user input may include defining at least one of allowed communication audio signal types, allowed audio formats, allowed delay values, and at least one acoustic modeling preference parameter.

[0033] The method may further include obtaining a communication audio signal type associated with the at least one spatial audio signal, and generating at least one output spatial audio signal from the at least one spatial audio signal and the communication audio signal based on the determined rendering method and an insertion point in the rendering process for the determined rendering method and / or a selection of rendering elements for the determined rendering method based on the rendering process parameters, and may include generating the at least one output spatial audio signal from the at least one spatial audio signal and the communication audio signal further based on the at least one communication audio signal type associated with the at least one spatial audio signal.

[0034] The rendering processes and / or rendering elements may include one or more of Doppler processing, direct sound processing, material filtering processing, early reflection processing, diffuse late reverberation processing, sound source extension processing, occlusion processing, diffraction processing, sound source transformation processing, externalization rendering, and in-head rendering.

[0035] Determining an insertion point in a rendering process for the determined rendering method and / or selecting a rendering element for the determined rendering method based on the rendering process parameters may include determining a rendering mode, the rendering mode including a value indicating the insertion point of the communication audio signal.

[0036] The value indicating the insertion point may include one of a first mode value indicating that the communication audio signal and the at least one spatial audio signal are inserted at the start of the rendering processing method, a second mode value indicating that the communication audio signal bypasses the rendering processing and is mixed directly with the output of the rendering processing applied to the at least one spatial audio signal, and a third mode value indicating that the rendering processing is fully applied to the at least one spatial audio signal while the communication audio signal is partially rendered.

[0037] A third mode value indicating that the communication audio signal has been partially rendered may be a value indicating that the communication audio signal is a direct sound rendering of a point sound source and a binaural rendering relative to a user position.

[0038] The method may further include determining an audio format type of the communication audio signal based on the rendering process parameters.

[0039] Determining an insertion point in a rendering process for the determined rendering method and / or selecting a rendering element for the determined rendering method based on the rendering process parameters may include determining an insertion point in a rendering process for the communication audio signal in the determined rendering method based on the audio format type.

[0040] Determining an insertion point in a rendering process of a communication audio signal in a determined rendering method based on the audio format type may include determining that if the communication audio signal has an audio format type of a pre-rendered spatial audio format, the insertion point in the rendering method is to be in direct mixing with the output of the rendering process applied to the at least one spatial audio signal.

[0041] According to a third aspect, there is provided an apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured by the at least one processor to cause the apparatus to at least: acquire at least one spatial audio signal for rendering within an immersive audio scene, acquire a communication audio signal and position information associated with the communication audio signal, acquire rendering process parameters associated with the communication audio signal, determine a rendering method based on the rendering process parameters, and determine an insertion point in the rendering process for the determined rendering method and / or a selection of rendering elements for the determined rendering method based on the rendering process parameters.

[0042] The apparatus may further be configured to generate at least one output spatial audio signal from the at least one spatial audio signal and the communication audio signal based on the determined rendering method and an insertion point in the rendering process for the determined rendering method and / or a selection of rendering elements for the determined rendering method based on the rendering process parameters.

[0043] The apparatus may be further adapted to determine at least one of an audio format, a tolerable delay value, and a communication audio signal delay associated with the communication audio signal.

[0044] An apparatus configured to determine an insertion point in a rendering process for a determined rendering method and / or a selection of rendering elements for the determined rendering method based on rendering process parameters may further be configured to perform at least one of: determining an insertion point in the rendering process further based on the determined at least one of an audio format, a permissible delay value, and a communication audio signal delay associated with the communication audio signal; and determining a rendering method and / or a selection of rendering elements for the determined rendering method based on the determined at least one of an audio format, a permissible delay value, and a communication audio signal delay associated with the communication audio signal.

[0045] The tolerable delay value may be the amount of delay permitted for utilizing the communication audio signal, and the communication audio signal delay may be a delay value determined based on the end-to-end delivery delay and the delay in rendering the communication audio.

[0046] The audio format associated with the communication audio signal may include one of a one-way communication audio signal and a conversational communication audio signal between users in an immersive scene.

[0047] An apparatus for generating at least one output spatial audio signal from at least one spatial audio signal and a communication audio signal based on a determined rendering method and an insertion point in a rendering process for the determined rendering method and / or a selection of rendering elements for the determined rendering method based on rendering process parameters may be configured to represent the communication audio signal as a high-order Ambisonic audio signal.

[0048] The apparatus may be further configured to obtain user input, and the apparatus configured to generate at least one output spatial audio signal from the at least one spatial audio signal and the communication audio signal based on the determined rendering method and an insertion point in the rendering process for the determined rendering method and / or a selection of rendering elements for the rendering method determined based on the rendering process parameters, is further configured to generate the at least one output spatial audio signal from the at least one spatial audio signal and the communication audio signal based on the user input, and the user input may be configured to define at least one of the allowed communication audio signal types, the allowed audio formats, the allowed delay values, and the at least one acoustic modeling preference parameter.

[0049] The apparatus may be configured to obtain a communication audio signal type associated with the at least one spatial audio signal, and the apparatus configured to generate at least one output spatial audio signal from the at least one spatial audio signal and the communication audio signal based on the determined rendering method and an insertion point in the rendering process for the determined rendering method and / or a selection of rendering elements for the determined rendering method based on the rendering process parameters may further be configured to generate the at least one output spatial audio signal from the at least one spatial audio signal and the communication audio signal based on the at least one communication audio signal type associated with the at least one spatial audio signal.

[0050] The rendering processes and / or rendering elements may include one or more of Doppler processing, direct sound processing, material filtering processing, early reflection processing, diffuse late reverberation processing, sound source extension processing, occlusion processing, diffraction processing, sound source transformation processing, externalization rendering, and in-head rendering.

[0051] An apparatus configured to determine an insertion point in a rendering process for a determined rendering method and / or a selection of rendering elements for the determined rendering method based on rendering process parameters may be configured to determine a rendering mode, the rendering mode comprising a value indicating an insertion point of the communication audio signal.

[0052] The value indicating the insertion point may include one of a first mode value indicating that the communication audio signal and the at least one spatial audio signal are inserted at the start of the rendering processing method, a second mode value indicating that the communication audio signal bypasses the rendering processing and is mixed directly with the output of the rendering processing applied to the at least one spatial audio signal, and a third mode value indicating that the rendering processing is fully applied to the at least one spatial audio signal while the communication audio signal is partially rendered.

[0053] A third mode value indicating that the communication audio signal is partially rendered may be a value indicating that the communication audio signal is a direct sound rendering of a point sound source and a binaural rendering relative to a user position.

[0054] The apparatus may be further characterized in that it determines an audio format type of the communication audio signal based on the rendering process parameters.

[0055] An apparatus for determining an insertion point in a rendering process of a determined rendering method and / or for selecting a rendering element of a determined rendering method based on rendering process parameters may be configured to determine an insertion point in a rendering process of a communication audio signal within a rendering method determined based on an audio format type.

[0056] An apparatus for determining an insertion point in a rendering process of a communication audio signal in a rendering method determined based on an audio format type may be configured to determine that if the communication audio signal has an audio format type of a pre-rendered spatial audio format, the insertion point in the rendering method is in direct mixing with the output of the rendering process applied to at least one spatial audio signal.

[0057] According to a fourth aspect, there is provided an apparatus comprising: means for obtaining at least one spatial audio signal for rendering within an immersive audio scene; means for obtaining a communication audio signal and position information related to the communication audio signal; means for obtaining rendering process parameters related to the communication audio signal; means for determining a rendering method based on the rendering process parameters; and means for determining an insertion point in the rendering process for the determined rendering method and / or a selection of rendering elements for the determined rendering method based on the rendering process parameters.

[0058] According to a fifth aspect, there is provided a computer program comprising instructions (or a computer readable medium comprising program instructions) for causing an apparatus to at least: obtain at least one spatial audio signal for rendering within an immersive audio scene; obtain a communication audio signal and position information associated with the communication audio signal; obtain rendering process parameters associated with the communication audio signal; determine a rendering method based on the rendering process parameters; and determine an insertion point in the rendering process for the determined rendering method and / or a selection of rendering elements for the determined rendering method based on the rendering process parameters.

[0059] According to a sixth aspect, there is provided a non-transitory computer-readable medium comprising program instructions to cause an apparatus to at least obtain at least one spatial audio signal for rendering within an immersive audio scene, obtain a communication audio signal and position information associated with the communication audio signal, obtain rendering process parameters associated with the communication audio signal, determine a rendering method based on the rendering process parameters, and determine an insertion point in the rendering process for the determined rendering method and / or a selection of rendering elements for the determined rendering method based on the rendering process parameters.

[0060] According to a seventh aspect, there is provided an apparatus comprising: an acquisition circuit configured to acquire at least one spatial audio signal for rendering within an immersive audio scene; an acquisition circuit configured to acquire a communication audio signal and position information associated with the communication audio signal; an acquisition circuit configured to acquire rendering process parameters associated with the communication audio signal; a decision circuit configured to determine a rendering method based on the rendering process parameters; and a decision circuit configured to determine an insertion point in the rendering process for the determined rendering method and / or a selection of rendering elements of the determined rendering method based on the rendering process parameters.

[0061] According to an eighth aspect, there is provided a computer-readable medium comprising program instructions for causing an apparatus to at least: obtain at least one spatial audio signal for rendering within an immersive audio scene; obtain a communication audio signal and position information associated with the communication audio signal; obtain rendering process parameters associated with the communication audio signal; determine a rendering method based on the rendering process parameters; and determine an insertion point in the rendering process for the determined rendering method and / or a selection of rendering elements for the determined rendering method based on the rendering process parameters.

[0062] An apparatus comprising means for performing the operations of the above method.

[0063] An apparatus configured to perform the operations of the above method.

[0064] A computer program comprising program instructions for causing a computer to carry out the above method.

[0065] A computer program product stored on the medium may cause an apparatus to perform the methods described herein.

[0066] The electronic device may include the apparatus described herein.

[0067] The chipset may include the devices described herein. [Brief description of the drawings]

[0068] For a better understanding of the present application, reference will now be made, by way of example, to the accompanying drawings, in which: [Figure 1] FIG. 1 is a schematic diagram of a suitable environment illustrating an example of the combination of virtual scene elements within a physical listening space. [Diagram 2]FIG. 2 is a schematic diagram of a system of devices for implementing an exemplary capture-to-rendering of an augmented reality scene, according to some embodiments. [Diagram 3] FIG. 3 shows a flow diagram of the operation of a system of devices such as that shown in FIG. 2, according to some embodiments. [Figure 4] FIG. 3 is a diagram that illustrates a schematic of an example of the renderer shown in FIG. 2, according to some embodiments. [Diagram 5] 5A-5C are schematic diagrams illustrating examples of the scene state derivation audio processor shown in FIG. 4 according to some embodiments. [Figure 6] 6 illustrates a flow diagram of operations within the exemplary scene state derivation audio processor shown in FIG. 5, according to some embodiments. [Figure 7] 1 illustrates a flow diagram of operations for implementing communication audio in a system according to some embodiments. [Figure 8] FIG. 8 illustrates a schematic diagram of an exemplary device suitable for implementing the illustrated apparatus. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0069] The following describes in more detail suitable devices and possible mechanisms for rendering augmented reality (AR) scene experiences and providing immersive audio communication processing capabilities.

[0070] As mentioned above, it is envisioned that "social VR" will be specified as a requirement for the MPEG-I6DoF audio standardization. The requirement is envisioned that the system will support the rendering of voice and audio from other users within the virtual environment. The voice and audio may be immersive. Additionally, in some embodiments, the apparatus and method are envisioned to support low-latency conversation between users within a given virtual environment. Additionally, the apparatus and method should support low-latency conversation between users within the given virtual environment and users outside the given virtual environment.

[0071] Additionally, the apparatus and method should enable audio and video synchronization of users and scenes, and further support metadata that specifies limitations and recommendations regarding the rendering of voice / audio from other users (e.g., regarding positioning and sound levels).

[0072] Thus, the embodiments described herein may implement low latency communication solutions such as, for example, 3GPP EVS / IVAS and interface with or implement MPEG-I6DoF rendering.

[0073] The differences between immersive and communication audio signals used in 6DoF scene rendering may lie in the decoding and rendering delays, different delivery mechanisms, and different usage constraints: Streaming or content delivery delays (such as those obtained with DASH) are expected or tolerated in 6DoF immersive audio delivery, since these are not low-latency delivery methods, whereas communication audio is usually interactive or requires low latency.

[0074] The rendering of communication audio in an immersive audio scene (MPEG Immersive Audio) is dynamic in nature, so the previously assumed rendering characteristics may not always be sufficient. This is because the delay budget for communication audio depends on several factors. One important factor is the use case (whether the communication is one-way commentary or two-way conversation, etc.), which requires different delay constraints depending on the use case. Another factor is the network conditions for communication audio, which may be different for different use cases within one virtual immersive audio usage session or different usage sessions. Changes in network conditions may result in different times for the actual rendering delay budget.

[0075] This can be considered similar to the AR situation where the usage scene is unknown, but the processing of communication audio is more dynamic and can change significantly over time. Thus, for communication audio, the delay budget (e.g., may be specified in the EIF or known via any higher-level module) can be known or estimated during content creation. However, the delay parameters of communication audio are almost always different (e.g., end-to-end delay, jitter, etc.) and unknown during content creation. In an embodiment of the present invention, the content creator delay budget parameter can be in a form that implicitly informs the budget by indicating the rendering mode of the communication audio. For example, a value of 0 indicates that the communication audio goes through the start of the entire immersive audio rendering pipeline, a value of 1 indicates that the communication audio bypasses the entire rendering pipeline and is mixed directly at the final stage, and a value of 2 indicates that the communication audio goes through minimal rendering (e.g., direct sound rendering of a point source and binaural rendering of a specified position).

[0076] The concepts described in further detail in the embodiments below relate to apparatus and methods configured to reliably render audio signals to a listener or user in the presence of various implementation-specific delay differences that are not significantly perceptible during streaming-based consumption of 6DoF scene data. These embodiments are configured to enable interactive usage scenarios to be implemented (as streaming-based consumption is less sensitive to delays than interactive consumption).

[0077] In the following description, the tolerable delay budget is the amount of delay that is tolerated for utilizing received communication audio. Typically, this is composed of the end-to-end delivery delay and the delay for rendering the communication audio in the 6DoF immersive audio scene. In the following, the term is used interchangeably with the tolerable delay value. Furthermore, in the following, network variations refer to changes in the end-to-end network delivery that can be based on delay, jitter, etc.

[0078] The embodiments described below relate to rendering of communication audio within an immersive audio scene with six degrees of freedom (i.e., the listener can move within the scene and the listener's position and pose are tracked), and an apparatus and method are described that are configured to ensure a desired (e.g., player, user, or content creator specified) rendering of received communication audio within an immersive audio scene, despite the dynamic nature of the communication audio due to network variations and immersive audio scene dependent characteristics. This may be achieved in some embodiments by deriving position information associated with the received communication audio (which may be, for example, a placeholder in the EIF indicating where to "place" the communication audio) from a bitstream associated with at least one spatial audio signal. In another embodiment, a communication audio placeholder may be added to any audio element, such as an audio object source, a channel source, or an HOA source. Audio element properties for each of the signal types (HOA, object, channel) may be added to such communication audio placeholder. In some embodiments, the apparatus and method may be further configured to obtain an allowed delay budget associated with the communication audio (e.g., obtained from the system, one-way or two-way, or from the EIF). Additionally, in some embodiments, the apparatus and method may be configured to obtain an audio format associated with the communication audio based on the allowed delay. The embodiments may be further configured to determine an appropriate insertion point in the rendering pipeline (in other words, derive where in the audio signal pipeline the communication audio is received for processing) based on the allowed delay, the communication audio format, the communication audio delay, and the audio rendering pipeline declaration.The apparatus and method according to some embodiments are further configured to determine at least one processing method and spatial rendering parameters for the communication audio based on at least one of the determined tolerable delay, the communication audio delay, and the audio format.

[0079] Thus, in some embodiments, the processing method relates to selecting communication audio processing depending on the signal type (e.g., whether the signal type is an object format signal, an HOA signal, or a channel format signal), and the rendering parameters may then refer to indicate one or more of the rendering approaches to be applied. For example, the rendering parameters may indicate that the rendering stage employs point source rendering of direct sounds and binaural rendering of audio objects.

[0080] In some further embodiments, the rendering parameters may be used to indicate or control other elements of the rendering process. For example, in some embodiments, the rendering parameters may be employed to indicate the selection or skipping of a rendering stage. Thus, in some embodiments, based on a certain insertion point in the rendering process operation, the start of the rendering follows a certain sequence that is dependent on the audio rendering pipeline stage sequence. However, the rendering parameters may be employed to indicate the skipping of some intermediate stages in the rendering process sequence.

[0081] In some embodiments where the communication audio signal is represented as a Higher Order Ambisonics (HOA) format, whether it is rendered as a single point HOA with internal transformation or as a 3DoF HOA depends on the allowed rendering delay. In some embodiments, the communication audio is implemented as audio objects or channels, and the amount of acoustic modeling performed on the audio objects depends on the network delay and the allowed utilization delay.

[0082] In some embodiments, the communication audio rendering is adapted according to preferences of the immersive audio scene player. These preferences include at least one of the following aspects: The types of communication audio allowed (e.g. one-way, conversational, between users using the same 6DoF scene, etc.) Allowed communication audio formats (e.g. audio objects, channels, 3DoF HOA, single HOA with transform) Allowable delay in rendering communication audio Acoustic Modeling Preferences

[0083] In some embodiments, depending on latency preferences, communication audio can be input and processed through a 6DoF audio rendering pipeline or mixed separately.

[0084] For example, if configured to place a communications audio signal into an immersive audio scene with maximum acoustic merging, but the delay budget dictates that certain features such as diffraction, occlusion, etc. are not possible, the renderer will determine the audio object path to minimize the occurrence of occlusion or diffraction.

[0085] Figure 2 shows an overview of an end-to-end AR / XR 6DoF audio system. In the example, three parts of the system are shown: a capture / generator device 201 configured to capture / generate and store / transmit audio information and associated metadata, and an augmented reality (AR) device 207 configured to output an appropriately processed audio signal based on the audio information and associated metadata. The example AR device 207 shown in Figure 2 has a 6DoF audio player 205 that retrieves and renders a 6DoF bitstream from a storage / distribution device 203.

[0086] In some embodiments as shown in FIG. 2, the capture / generator device 201 comprises an Encoder Input Format (EIF) generator 211. The Encoder Input Format (EIF) generator 211 (or more generally a scene definer) is configured to define a 6DoF audio scene. In some embodiments, the scene may be described by EIF (Encoder Input Format) or any other suitable 6DoF scene description format. EIF also refers to audio data that constitutes the audio scene. The Encoder Input Format (EIF) generator 211 is configured to create EIF (Encoder Input Format) data, which is a scene description of the content creator. The scene description information includes virtual scene geometry information such as the positions of audio elements. In addition, the scene description information may include other relevant metadata such as directivity, size, and other acoustically relevant elements. For example, the relevant metadata may include the positions of virtual walls and their acoustic properties as well as other acoustically relevant objects such as occlusions. Examples of acoustic properties are acoustic material properties such as (frequency-dependent) absorption or reflection coefficients, the amount of scattered energy, or transmission properties. In some embodiments, the virtual acoustic environment may be described according to its (frequency-dependent) reverberation time or diffuse-to-direct sound ratio. The EIF generator 211 in some embodiments is more commonly known as a virtual scene information generator. The EIF parameters 214 may be provided to a suitable (MPEG-I) encoder 217 in some embodiments.

[0087] In some embodiments, the capture / generator device 201 comprises an audio content generator 213. The audio content generator 213 is configured to generate audio content corresponding to the audio scene. The audio content generator 213 in some embodiments is configured to generate or otherwise acquire audio signals related to the virtual scene. For example, in some embodiments, these audio signals may be acquired or captured using a suitable microphone or array of microphones, may be based on processed captured audio signals, or may be synthesized. In some embodiments, the audio content generator 213 is further configured to generate or acquire audio parameters related to the audio signals, such as the position within the virtual scene, the directionality of the signal, etc. The audio signals and / or parameters 212 may be provided to a suitable (MPEG-I) encoder 217 in some embodiments.

[0088] In some embodiments, the capture / generator device 201 includes a communication audio processing data generator 215. The communication audio processing data generator 215 is configured to generate information carried in the content creator bitstream to indicate what type of communication audio (e.g., interactive, one-way, etc.) is allowed for this particular immersive audio scene. For example, some content creators may allow incoming communication audio from any caller, only allow communication audio from other users using the same 6DoF audio content, and allow communication audio between any two users using any 6DoF audio content. Additionally, the content creator bitstream carries information about which rendering stages are allowed and which are not allowed.

[0089] In some implementations, the communication audio processing parameters may depend on device profile preferences, application settings, or user preference settings.

[0090] For example, in some embodiments, the parameters may be implemented in a structure such as ObjectSourceCAStruct(). The ObjectSourceCAStruct() structure is an extension of the audio object metadata. In some embodiments, this structure may appear as a structure in the audio object metadata. The following examples discuss audio objects, but may similarly be extended to communication audio structures for HOAs and channels.

[0091] aligned(8) ObjectSourceCAStruct(){ unsigned int(16) object_audio_identifier; / / object audio index unsigned int(1) ca_prototype_flag; / / commmunication audio prototype flag unsigned int(1) active; / / active or inactive flag unsigned int(1) hasExtent; unsigned int(32) gainDB; unsigned int(32) referenceDistance; bit(5) reserved = 0; if(ca_prototype_flag) { unsigned int(1) exclude_clustering_flag; / / communication audio is excluded from clustering bit(7) reserved = 0; CommunicationAudioIngestionStruct(); DynamicIndexStruct(); } else { MPEGHDecodedAudioIndex; / / index to obtain MPEG-H encoded audio stream Location(); if(hasExtent) ExtentStruct(); } } aligned(8) DynamicIndexStruct(){ unsigned int(16) stream_identifier; / / dynamic ID allocated by renderer / player for the communication audio } aligned(8) Location(){ signed int(32) pos_x; signed int(32) pos_y; signed int(32) pos_z; signed int(32) orient_yaw; signed int(32) orient_pitch; signed int(32) orient_roll; unsigned int(1) cspace; / / with respect to listening space origin if 1 with respect to user if 0 bit(7) reserved = 0; }

[0092] ca_prototype_flag being 1 indicates to the player that it should prepare to receive communication audio. Communication audio ingestion related information is described by CommunicationAudioIngestionStruct(), which also has information about the type(s) of communication audio that are tolerated or allowed for 6DoF audio scenes. Furthermore, there is a flag exclude_clustering_flag that indicates whether clustering is possible for communication audio or not. In the absence of this flag, clustering is disabled by default. These communication audio types can be two-way, one-way, two-way, between users using the same content, between users using different content. The ingestion structure also carries information about required (RequiredRenderingStagesStruct()) and disallowed (DisallowedRenderingStagesStruct()) rendering stages. Furthermore, rendering_modes can also present in a compact way the need for communication audio to be processed through the immersive audio rendering pipeline or to be completely bypassed. In the latter case, the communication audio is rendered outside the immersive rendering pipeline and mixed with the output of the immersive audio content rendering pipeline. If the rendering_modes_present flag value is absent or equal to 0, rendering is performed according to the communication audio signal element properties. Typically, when the ca_rendering_modes_present value is 1, other data structures such as RequiredRenderingStagesStruct(), DisallowedRenderingStagesStruct(), ca_rendering_max_latency, etc. do not need to be present.

[0093] aligned(8) CommunicationAudioIngestionStruct(){ unsigned int(1) ca_co_conversational_allowed; / / bidirectional call with another user in the same 6DoF immersive audio scene unsigned int(1) ca_co_oneway_allowed; / / commentary from another user in the same 6DoF immersive audio scene unsigned int(1) ca_conversational_allowed; / / bidirectional call unsigned int(1) ca_oneway_allowed; / / commentary if(ca_co_conversational_allowed){ unsigned int(1) ca_delay_threshold_present; unsigned int(1) required_stages_present; unsigned int(1) disallowed_rendering_stages_present; unsigned int(1) ca_rendering_modes_present; bit(4) reserved = 0; if(ca_delay_threshold_present) unsigned int(32) ca_rendering_maxlatency; if(required_stages_present) RequiredRenderingStagesStruct(); if(disallowed_stages_present) DisallowedRenderingStagesStruct(); if(ca_rendering_modes_present) unsigned int(8) rendering_modes_type; } if(ca_co_oneway_allowed){ unsigned int(1) ca_delay_threshold_present; unsigned int(1) required_stages_present; unsigned int(1) disallowed_rendering_stages_present; unsigned int(1) ca_rendering_modes_present; bit(4) reserved = 0; if(ca_delay_threshold_present) unsigned int(32) ca_rendering_maxlatency; if(required_stages_present) RequiredRenderingStagesStruct(); if(disallowed_stages_present) DisallowedRenderingStagesStruct(); if(ca_rendering_modes_present) unsigned int(8) rendering_modes_type; } if(ca_oneway_allowed){ unsigned int(1) ca_delay_threshold_present; unsigned int(1) required_stages_present; unsigned int(1) disallowed_rendering_stages_present; unsigned int(1) ca_rendering_modes_present; bit(4) reserved = 0; if(ca_delay_threshold_present) unsigned int(32) ca_rendering_maxlatency; if(required_stages_present) RequiredRenderingStagesStruct(); if(disallowed_stages_present) DisallowedRenderingStagesStruct(); if(ca_rendering_modes_present) unsigned int(8) rendering_modes_type; } if(ca_conversational_allowed){ unsigned int(1) ca_delay_threshold_present; unsigned int(1) required_stages_present; unsigned int(1) disallowed_rendering_stages_present; unsigned int(1) ca_rendering_modes_present; bit(4) reserved = 0; if(ca_delay_threshold_present) unsigned int(32) ca_rendering_maxlatency; if(required_stages_present) RequiredRenderingStagesStruct(); if(disallowed_stages_present) DisallowedRenderingStagesStruct(); if(ca_rendering_modes_present) unsigned int(8) rendering_modes_type; } } aligned(8) RequiredRenderingStagesStruct(){ unsigned int(8) num_stages; for(i=0;i <num_stages;i++){ unsigned int(8) rendering_stage_idx; } } aligned(8) DisallowedRenderingStagesStruct(){ unsigned int(8) num_stages; for(i=0;i <num_stages;i++){ unsigned int(8) rendering_stage_idx; } }

[0094] [Table 1]

[0095] [Table 2]

[0096] In an embodiment, in addition to the rendering stage, the MPEG-H decoding delay is also taken into account to determine the tolerable delay threshold for communication audio rendering in an MPEG-I immersive audio scene. The decoding delay may depend on the audio format of the audio elements in the MPEG-I immersive audio scene.

[0097] In some embodiments, the capture / generator device 201 comprises an encoder 217. The encoder is configured to receive the EIF parameters 212, the communication audio processing parameters 216, and the audio signal / audio parameters 214 and decode them to generate an appropriate bitstream.

[0098] The encoder 217 can use, for example, the EIF parameters 212, the communication audio processing parameters 216, and the audio signal / audio parameters 214 to generate MPEG-I 6DoF audio scene content that is stored in a format that may be suitable for streaming over a network. The distribution can be in any suitable format, such as MPEG-DASH (Dynamic Adaptive Streaming Over HTTP), HLS (HTTP Live Streaming), etc. The 6DoF bitstream carries the MPEG-H encoded audio content and the MPEG-I 6DoF bitstream. The content creator bitstream generated by the encoder based on the EIF and audio data can be formatted and encapsulated in a manner similar to MHAS packets (MPEG-H 3D audio streams). The encoded bitstream in some embodiments is passed to a suitable content storage module 219. For example, as shown in FIG. 2, the encoded bitstream is passed to the MPEG-I 6DoF content storage module 219. In this example, the encoder 217 is located within the capture / generator device 201 , however, it will be appreciated that the encoder 217 may be separate from the capture / generator device 201 .

[0099] In some embodiments, the capture / generator device 201 has a content storage module. For example, as shown in FIG. 2, the encoded bitstream is passed to an MPEG-I6DoF content storage 219 module. In such an embodiment, the audio signal is transmitted in a separate data stream from the encoded parameters. In some embodiments, the audio signal and parameters are stored / transmitted as a single data stream or format.

[0100] The content storage 219 is configured to store and provide content (including EIF-derived content creator bitstreams with communication audio processing parameters) to the AR device 207.

[0101] In some embodiments, the AR device 207 having a head mounted device (HMD) is a playback device for AR use of 6DoF audio scenes.

[0102] The AR device 207 in some embodiments has at least one AR sensor 221. The at least one AR sensor 221 may include a multi-modal sensor such as a visual camera array, a depth sensor, LiDAR, etc. The multi-modal sensor is used by the AR-utilizing device to generate information of the listening space. This information may include material information, objects of interest, etc. This sensor information may be passed to an AR processor 223 in some embodiments.

[0103] The AR device 207 in some embodiments includes at least one position / orientation sensor 227. The at least one position / orientation sensor 227 may include any suitable sensor or sensors configured to determine the position and / or orientation of the listener within the physical listening space. For example, the sensors may include digital compass / gyroscopes, positioning beacons, etc. In some embodiments, the sensors employed in the AR sensor 221 are further used to determine the orientation and / or orientation of the listener. This sensor information may be passed to the renderer 235 in some embodiments.

[0104] In some embodiments, the AR device 207 includes an input configured to receive the communication audio 200 .

[0105] Further, in some embodiments, the AR device 207 has a communication audio controller 253. The communication audio controller 253 is configured to output control information to the renderer 235 and control the integration of the communication audio 200 and the interactive audio (which may be part of the bitstream 220). The communication audio controller 253 in some embodiments is configured to generate information in a format described below or any other suitable format. In some embodiments, this information is in the form of a rendering stage declaration that can signal a desired order so that any of the processing stages can be skipped without causing repetitive processing or undesirable output for subsequent rendering processes. The rendering pipeline declaration in some embodiments is available to the scene management processor. An exemplary structure for the information can be as follows:

[0106] aligned(8) RenderingStagesInfoStruct(){ unsigned int(8) num_stages; for(i=0;i <num_stages;i++){ unsigned int(8) render_stage_idx; unsigned int(32) mean_delay_value; unsigned int(32) sd_delay_value; unsigned int(8) input_audio_type; } }

[0107] In some embodiments, the rendering stages of the sonification pipeline are listed, with the values ​​for mean_delay_value and sd_delay_value being -1 if unavailable.

[0108] In some embodiments, the communication audio controller 253 is configured to generate control information such as the format, delivery delay, and jitter of the communication audio. In some embodiments, this information can be predefined values ​​stored in the renderer, so that if the controller 253 is not present or does not provide this information, the renderer is configured to use default values. For example, the delay value of a class type can be a predefined default value used by the renderer. In some embodiments, the control information can be passed in the following structure:

[0109] aligned(8) CommunicationAudioInfoStruct(){ unsigned int(4) ca_class_type; unsigned int(4) ca_format_type; unsigned int(32) ca_delivery_latency; }

[0110] [Table 3]

[0111] [Table 4]

[0112] If the comms audio format is 0 or 1, it is input as comms audio by the immersive audio rendering pipeline. However, in the case of pre-rendered spatial audio, the comms audio is input directly to the mixer block.

[0113] In another embodiment, communication audio processing is embedded within the audio element metadata specified for objects, channels, and HOA sources. As a result, communication audio can be rendered by an MPEG-I renderer as any other audio element with communication audio specific properties. These communication audio specific rendering preferences are specified as control data for selecting and / or rejecting one or more rendering stages.

[0114] aligned(8) ObjectSourceStruct(){ unsigned int(16) index; / / object audio index unsigned int(1) ca_flag; / / placeholder for rendering communication audio as audio object unsigned int(1) active; / / active or inactive flag unsigned int(1) hasExtent; unsigned int(32) gainDB; unsigned int(32) referenceDistance; bit(5) reserved = 0; if(ca_flag) { CommunicationAudioRenderingStruct(); unsigned int(16) CommunicationAudioIndex; / / Identifier to receive the communication audio stream } else { unsigned int(16) MPEGHDecodedAudioIndex; / / index to obtain MPEG-H encoded audio stream Location(); / / index to obtain MPEG-H encoded audio stream } if(hasExtent) ExtentStruct(); } aligned(8) CommunicationAudioRenderingStruct(){ unsigned int(1) rendering_modes_present; unsigned int(1) dynamic_modes_; if(rendering_modes_present) { unsigned int(8) rendering_modes; } bit(7) reserved = 0; }

[0115] The ObjectSourceStruct(), ObjectSourceCAStruct(), or any of their configuration parameters or structures may change over the duration of an audio scene. As a result, communication audio in some embodiments may be allowed or disallowed depending on the prevailing metadata information. Additionally, the rendering mode or insertion point may change over the duration of an audio scene.

[0116] In yet another embodiment, the communication audio metadata, for example the audio element metadata mentioned above that is a placeholder for communication audio, carries an indication flag communicationAudioRenderImmediateFlag.

[0117] If communicationAudioRenderImmediateFlag==0, communication audio is rendered or mixed immediately into the rendered immersive audio scene without additional delay.

[0118] If communicationAudioRenderImmediateFlag==1, communication audio is rendered according to the rendering metadata and properties specified on the audio element.

[0119] Metadata for communication audio can also be delivered as dynamic updates to the renderer. Communication audio can be delivered as a new MHAS packet with PACTYP_CAAUDIODATA, and labels can be used to indicate the corresponding metadata and the dynamic update metadata that applies to PACTYP_CAAUDIODATA. The PACTYP_CAAUDIODATA packet carries a payload of ObjectSourceStruct(), HOASourceStruct(), ChannelSourceStruct(), or a subset of these structures, along with communication audio rendering or capture parameters.

[0120] In one implementation, PACTYP_CAAUDIODATA carries audio data in the form of PCM. As a result, PACTYP_CAAUDIODATA is followed by PACTYP_PCMCONFIG and PACTYP_PCMDATA. The preceding PACTYP_CAAUDIODATA packet allows the renderer to identify the PCM data that corresponds to the communication audio data.

[0121] In some embodiments, the AR device 207 has a suitable output device, which in the example shown in Figure 2 is shown as headphones 241 configured to receive the spatial audio output 240 generated by the renderer 235, although any suitable output transducer may be arranged.

[0122] In some embodiments, the AR device 207 includes a player / renderer apparatus 205. The player / renderer apparatus 205 is configured to receive a bitstream including the EIF-derived content creator bitstream 220, AR sensor information, user position and / or attitude information, communication audio 220, and control information from a communication audio controller, and to determine from this information an appropriate spatial audio output 240 (which may be incorporated within the AR device 207) that can be passed to an appropriate output device, shown in FIG. 2 as headphones 241.

[0123] In some embodiments, the player / renderer device 205 includes an AR processor 223. The AR processor 223 is configured to receive sensor information from at least one AR sensor 221 and generate appropriate AR information that can be passed to the LSDF generator 225. For example, in some embodiments, the AR processor is configured to perform fusion of the sensor information from each of the sensor types.

[0124] In some embodiments, the player / renderer device 205 includes a listening space description file (LSDF) generator 225. The listening space description file (LSDF) generator 225 is configured to receive the output of the AR processor 223 and generate a listening space description for AR use from information obtained from the AR sensing interface. The format of the listening space can be any suitable format. The LSDF format can be used to create the LSDF. This description conveys listening space or room information including acoustic properties (e.g., a mesh enclosing the listening space including the material of the mesh surface), spatially variable elements of the scene, called anchors in the listening space description. The LSDF generator is configured to output this listening scene description information to the renderer 235.

[0125] In some embodiments, the player / renderer device 205 comprises a receive buffer 231 configured to receive the content creator bitstream (including EIF information) 220. The buffer 231 is configured to pass the received data and pass the data to a decoder 233.

[0126] In some embodiments, the player / renderer device 205 has a decoder 233 configured to obtain the encoded bitstream from the buffer 231 and output the decoded EIF information and the communicated audio processing parameters (together with the decoded audio data if in the same data stream) to the renderer 235.

[0127] In some embodiments, the player / renderer device 205 includes a communication receiver buffer and decoder 251 configured to receive the communication audio 200 and decode the encoded audio data to pass it to the renderer 235.

[0128] In some embodiments, the player / renderer device 205 comprises a renderer 235 configured to receive the decoded EIF information (including the decoded immersive audio data if in the same data stream), listening scene description information, listener position and / or attitude information, decoded communication audio, and communication audio control information. The renderer 235 is configured to generate spatial audio output signals and pass these to an output device, as shown in FIG. 2 by spatial audio output 240 to headphones 241.

[0129] With reference to FIG. 3, an example of the operation of the system shown in FIG. 2 is shown.

[0130] As shown in FIG. 3, in step 301, communication audio processing data is obtained (or generated).

[0131] The method may include generating or alternatively obtaining EIF information, via step 303, as shown in FIG.

[0132] Further, as shown in FIG. 3, audio data is obtained (or generated) via step 305 .

[0133] Then, as shown in FIG. 3, step 307 encodes the EIF information, the communication audio processing data, and the audio data.

[0134] The encoded data is then stored / retrieved or transmitted / received, per step 309, as shown in FIG.

[0135] Further, as shown in FIG. 3, AR scene data is acquired by step 311 .

[0136] As shown in FIG. 3, step 313 generates listening space description (file) information from the detected AR scene data.

[0137] Further, as shown in FIG. 3, communication audio control information may be obtained, via step 312 .

[0138] Also shown in FIG. 3, communication audio data is obtained, via step 314 .

[0139] Additionally, as shown in FIG. 3, position and / or attitude data of the listener / user may be obtained, via step 315.

[0140] A spatial audio signal may then be rendered based on the audio data, the communication audio control information, the communication audio, the EIF information, the LSDF data, and the position and / or attitude data. Specifically, the rendering includes a combination of the audio signals, as shown in FIG. 3 at step 317.

[0141] After rendering the spatial audio signals, they may be output to an appropriate output device, such as headphones, by step 319, as shown in FIG.

[0142] FIG. 4 illustrates an example renderer 235 suitable for implementing some embodiments and may be configured to provide a fused audio signal.

[0143] For example, Figure 4 shows that in front of the renderer 235 is a bitstream parser 401 configured to receive the decoded 6DoF bitstream. The parsed EIF data can then be passed to a scene manager / processor 403.

[0144] The renderer 235 in some embodiments includes a scene manager / processor 403 configured to receive the parsed EIF from the bitstream parser 401, communication control information, such as delay, jitter, and parameters defining the format of the communication audio data 402.

[0145] In some embodiments, the scene manager / processor 403 includes a communication audio adaptation processor 411 configured to control the auralization pipeline (or DSP processing for rendering) according to information such as content creator preferences (obtained from the bitstream) regarding communication audio processing, communication audio tolerance budget for delay, etc.

[0146] The scene management information may then be passed to the scene state derivation audio processor 405.

[0147] The scene manager / processor 403 may be further configured to take the decoded 6DoF audio signal, the processed scene information, and the position and / or pose of the listener, and generate therefrom a spatial audio signal output. As indicated above, the effect of the scene manager / processor 403 is such that any known or suitable spatial audio processing implementation may be employed (the auralization pipeline is not involved in any prior scene processing).

[0148] The renderer 235 in some embodiments includes a scene state derived audio processor (DSP processing and auralization) 405 configured to receive configuration information from the scene manager / processor 403, the decoded immersive audio (MPEG-I audio / decoded MPEG-H audio) 400, and the decoded communication audio signal 450 to generate a spatial audio signal.

[0149] With respect to FIG. 5, an exemplary scene state derivation audio processor (DSP processing and auralization) 405 is illustrated. The scene state derivation audio processor is configured to obtain decoded immersive audio (IA) and communication audio (CA) flows through different rendering modules in the auralization pipeline. In some embodiments, the processor 405 is configured to employ multiple processing paths based on the IA format type (IA1 and IA2). Similarly, the communication audio (CA) may have multiple paths (in this example, there are three candidate paths, CA1, CA2, and CA3, according to the CA format). The different rendering modules are annotated with index numbers indicating possible insertion points for IA or CA. Furthermore, the exemplary processor 405 illustrated herein has two output options: a first option (O2) with a mixer and another option (O1) without a mixer.

[0150] The first path IA1-CA1 is adapted to apply the following processing operations to the audio signal and to apply audio object modelling operations:

[0151] In some embodiments, the scene state derived audio processor (DSP processing and auralization) 405 has a Doppler processor 501 configured to control Doppler processing of immersive audio (IA) and communications audio (CA) based on control information from setting information from the scene manager / processor 403.

[0152] In some embodiments, the scene state derivation audio processor (DSP processing and auralization) 405 has a direct sound processor 503 configured to control direct sound processing of immersive audio (IA) and communications audio (CA) based on control information of the setting information from the scene manager / processor 403.

[0153] In some embodiments, the scene state derivation audio processor (DSP processing and auralization) 405 has a material filter processor 505 configured to control material filtering of immersive audio (IA) and communication audio (CA) based on control information of the setting information from the scene manager / processor 403.

[0154] In some embodiments, the scene state derivation audio processor (DSP processing and auralization) 405 has an early reflection processor 507 configured to control early reflection sound processing of immersive audio (IA) and communications audio (CA) based on control information from setting information from the scene manager / processor 403.

[0155] In some embodiments, the scene state derivation audio processor (DSP processing and auralization) 405 has a late reverberation processor 509 configured to control late reverberation processing of immersive audio (IA) and communication audio (CA) based on control information from setting information from the scene manager / processor 403.

[0156] In some embodiments, the scene state derivation audio processor (DSP processing and auralization) 405 has an extended processor configured to control extended sound processing of immersive audio (IA) and communications audio (CA) based on control information from configuration information from the scene manager / processor 403.

[0157] It will be appreciated that the order of the above processes may be in any suitable order. The output of the object processing processor may be passed to a higher order Ambisonics or spatializer processor 541.

[0158] The second path IA2-CA2 applies the following processing operations to the audio signal, which may apply higher order Ambisonics processing:

[0159] In some embodiments, the scene state derivation audio processor (DSP processing and auralization) 405 has a SP Higher Order Ambisonics Processor 521 configured to control SP Higher Order Ambisonics processing of immersive audio (IA) and communications audio (CA) based on control information from configuration information from the scene manager / processor 403.

[0160] In some embodiments, the scene state derivation audio processor (DSP processing and auralization) 405 has an MP Higher Order Ambisonic Processor 523 configured to control MP Higher Order Ambisonic processing of immersive audio (IA) and communications audio (CA) based on control information from setting information from the scene manager / processor 403.

[0161] The output of the higher order Ambisonics processing may, in some embodiments, be passed to a higher order Ambisonics spatializer processor 541.

[0162] The first and second paths go to the high order Ambisonics of the spatializer processor 541 and are configured to select either one of the two paths to output as output O1 or to the mixer 551 based on the audio format, such as control information from the setting information from the scene manager / processor 403.

[0163] The third path CA3 may apply the following processing operations to the audio signal, and may apply a rendering process.

[0164] In some embodiments, the scene state derivation audio processor (DSP processing and auralization) 405 has a communication audio output rendering audio processor 531 configured to control the processing of communication audio (CA) based on control information from setting information from the scene manager / processor 403.

[0165] The output of the communication audio output rendering audio processor 531 may, in some embodiments, be passed to a mixer 551.

[0166] In some embodiments, the scene state derivation audio processor (DSP processing and audibility) 405 has a mixer configured to receive the outputs of the communication audio output rendering audio processor 531 and the high order Ambisonics or spatializer processor 541 and mix them based on control information from the setting information from the scene manager / processor 403 to generate a mixed output O2.

[0167] An exemplary flow diagram illustrating the operation of the scene state derivation audio processor (DSP processing and auralization) 405 as shown in some embodiments is shown in FIG.

[0168] A decoded immersive audio input is received or obtained from a listening space description, by step 601, as shown in FIG.

[0169] Further, a communication audio input is received or acquired by step 603 as shown in FIG.

[0170] Additionally, a processing pipeline is shown.

[0171] The spatialization processing pipeline is shown by steps 605 to 615. These are Doppler processing shown in step 605 of Fig. 6, direct sound processing shown in step 607 of Fig. 6, material filtering processing shown in step 609 of Fig. 6, early reflection processing shown in step 611 of Fig. 6, late reverberation processing shown in step 613 of Fig. 6, and extension processing shown in step 615 of Fig. 6.

[0172] The Ambisonics processing pipeline is illustrated by steps 621 and 623. These are SP higher order Ambisonics processing, illustrated in Figure 6 by step 621, and MP higher order Ambisonics processing, illustrated in Figure 6 by step 623.

[0173] The pre-rendered communication audio output pipeline is indicated by the communication audio output rendering audio process by step 631 as shown in FIG.

[0174] Further, as shown in FIG. 6, step 625 presents the selected spatialized or Ambisonics pipeline output.

[0175] In some embodiments, the output of the selected spatialized or Ambisonics processing pipeline, in other words the output of the immersive audio rendering pipeline, is then output as an unmixed audio signal by step 627, as shown in FIG. 6.

[0176] In some embodiments, the selected spatialized or Ambisonics processing pipeline output and the pre-rendered communications audio output pipeline are mixed and output by step 641 as shown in FIG. 6.

[0177] FIG. 7 is a flow diagram illustrating a procedure for determining appropriate audio insertion points and selection of processing steps for rendering communication audio.

[0178] As shown in Fig. 7, the method is configured by step 701 to receive a content creator bitstream for communication audio processing, which may be obtained as part of an MPEG-I6DoF content.

[0179] A check is then made to determine whether the currently active immersive audio scene allows the use of communication audio, which is shown in FIG.

[0180] If usage is permitted and supported, the next action is configured to retrieve communication audio information, as shown in FIG.

[0181] A further check may then be performed to determine whether the permitted communication audio type (e.g., description, interactivity, etc.) is supported, per step 707, as shown in Figure 7. In some embodiments (not shown) where the check determines that the audio type is not permitted, the method should pause the immersive audio scene and continue or switch the communication audio.

[0182] If the type is supported, then a rendering stage declaration and associated delay information is received or otherwise obtained, as shown in Figure 7, by step 709. This provides information about available rendering stages and potential insertion points for the communication audio.

[0183] In some embodiments, as shown in FIG. 7, a further checking operation may follow to determine whether delay threshold information is present, per step 711.

[0184] If delay threshold information is present, the method may be configured to utilize the delay of the rendering stage to select a stage from the declared rendering pipeline, as shown in FIG.

[0185] In some embodiments, this is subtracting the communication audio delay (ca_delivery_latency) from the delay threshold to obtain an updated delay requirement; determining candidate rendering stages applicable to the format type (ca_format_type) using the communication audio format type; If DisallowedRenderingStagesStruct() is present, its information can be used to discard stages from the associated declared rendering pipeline, Prioritize the inclusion of rendering stages represented in RequiredRenderingStagesStruct() while complying with the latest laziness requirements; and Controlling the insertion or input of communication audio in the first rendering stage obtained by the above operations; It can be implemented by:

[0186] If the delay threshold information is not present, the method may be configured to select a stage from the declared rendering pipeline using the required and disallowed stage information from the content creator bitstream. Furthermore, if a rendering mode is present, the rendering is set to be performed based on the rendering mode. The selection of the stage and the setting of the rendering based on the rendering mode, if present, is shown in FIG. 7 at step 715. This may, for example, use the following operations: Obtaining the latest latency requirement based on the requirement derived by taking the difference between ca_delivery_latency and ca_class_type; determining, based on the communication audio format type, possible rendering stages applicable to the ca_format_type; DisallowedRenderingStagesStruct(), if present, is used to discard stages from the associated declared rendering pipeline, Prioritize including rendering stages represented by RequiredRenderingStagesStruct() while complying with the latest laziness requirements, and Inserting communication audio into the first rendering stage obtained by the above operation.

[0187] The communication audio is then rendered, via step 717, as shown in FIG.

[0188] In some embodiments, a new communication audio delay is then obtained. If the difference from the current estimated delay requirement changes by more than a predetermined threshold, the audio rendering pipeline is modified. This is shown in Figure 7, where a check step is performed by step 719 to determine whether there has been a change in the communication audio information, and if the answer is yes, operation returns to step 705.

[0189] An example of a communication audio scenario for audio object formats is treating a communication audio signal whose format type is mono as an audio object if the content creator's bitstream specified similar acoustic modeling as an audio object. The audio object is rendered with the specified acoustic processing steps in the rendering pipeline as long as the latency requirements are met. However, in another example, if a particular rendering processing step (e.g., sound source extensions, or such steps that add significant rendering latency) is observed, that particular rendering step is omitted unless it is part of the required stage metadata.

[0190] A further example is HOA format communication audio where communication audio is delivered as an HOA source with a content creator bitstream indicating the HOA source with translation support within the HOA source. It is determined that if the rendering pipeline is selected to include single HOA source rendering, it can accommodate this processing based on delay constraints. In another example of the same scene, it is determined that the communication audio delay is too large to allow single point HOA rendering with conversion. As a result, the communication audio of the HOA source is rendered without conversion processing, but is mixed directly with the immersive audio output in the mixing block.

[0191] 8 is an exemplary electronic device that may represent any of the devices described above. The device may be any suitable electronic device or device. For example, in some embodiments, device 1400 may be a mobile device, a user equipment, a tablet computer, a computer, an audio playback device, etc.

[0192] In some embodiments, the device 1400 includes at least one processor or central processing unit 1407. The processor 1407 may be configured to execute various program code, such as methods as described herein.

[0193] In some embodiments, the apparatus 1400 comprises a memory 1411. In some embodiments, at least one processor 1407 is coupled to the memory 1411. The memory 1411 may be any suitable storage means. In some embodiments, the memory 1411 comprises a program code section for storing program code implementable in the processor 1407. Furthermore, in some embodiments, the memory 1411 may further comprise a storage data section for storing data, e.g., data that has been processed or is to be processed according to the embodiments as described herein. The implementation program code stored in the program code section and the data stored in the storage data section may be retrieved by the processor 1407 whenever needed via the memory-processor coupling.

[0194] In some embodiments, the device 1400 comprises a user interface 1405. The user interface 1405 may be coupled to a processor 1407 in some embodiments. In some embodiments, the processor 1407 may control the operation of the user interface 1405 and receive input from the user interface 1405. In some embodiments, the user interface 1405 may allow a user to input instructions to the device 1400, for example, via a keypad. In some embodiments, the user interface 1405 may allow a user to obtain information from the device 1400. For example, the user interface 1405 may have a display configured to display information from the device 1400 to the user. The user interface 1405 may be configured with a touch screen or touch interface in some embodiments that can both allow information to be input into the device 1400 and also display information to a user of the device 1400. In some embodiments, the user interface 1405 may be a user interface for communicating with a position determining device as described herein.

[0195] In some embodiments, the apparatus 1400 comprises an input / output port 1409. The input / output port 1409 in some embodiments comprises a transceiver. The transceiver in such embodiments may be coupled to the processor 1407 and configured to enable communication with other apparatuses or electronic devices, for example, via a wireless communication network. The transceiver or any suitable transceiver or transmitting and / or receiving means may in some embodiments be configured to communicate with other electronic devices or apparatuses via a wire or wired coupling.

[0196] The transceiver may communicate with the further device by any suitable known communication protocol, for example in some embodiments the transceiver may use a suitable Universal Mobile Telecommunications System (UMTS) protocol, a Wireless Local Area Network (WLAN) protocol such as IEEE 802.X, a suitable short range radio frequency communication protocol such as Bluetooth, or an Infrared Data Path (IRDA).

[0197] The transceiver input / output port 1409 may be configured to receive signals and, in some embodiments, determine parameters as described herein using a processor 1407 executing appropriate code.

[0198] Furthermore, although exemplary embodiments have been described above, it is pointed out herein that several variations and modifications are possible to the disclosed solutions without departing from the scope of the present invention.

[0199] In general, various embodiments may be implemented in hardware or special purpose circuits, software, logic, or any combination thereof. Some aspects of the present disclosure may be implemented in hardware, and other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the present disclosure is not limited thereto. Although various aspects of the present disclosure may be illustrated and described as block diagrams, flow charts, or using some other graphical representation, it will be appreciated that these blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing device, or combinations thereof, as non-limiting examples.

[0200] As used herein, the term "circuitry" may refer to one or more or all of the following: (a) a hardware-only circuit implementation (e.g., an implementation in only analog and / or digital circuitry); and (b) A combination of hardware circuitry and software, such as (where applicable) (i) a combination of analog and / or digital hardware circuitry and software / firmware; and (ii) software (including digital signal processors), software, and hardware processor portions having memory that work together to cause a device, such as a mobile phone or a server, to perform various functions; and (c) Hardware circuits and processors, such as microprocessors or parts of microprocessors, that require software (e.g., firmware) to operate, but the software may be absent when not necessary for operation.

[0201] This definition of circuitry applies to all uses of the term in this application, including any claims. As a further example, as used in this application, the term circuitry also covers merely a hardware circuit or processor (or processors) or portions of a hardware circuit or processor and its (their) accompanying software and / or firmware implementations.

[0202] The term circuitry also covers, for example, baseband or processor integrated circuits for mobile terminals, or similar integrated circuits in servers, cellular network devices, or other computing or network devices, where applicable to particular claim elements.

[0203] The embodiments of the present disclosure may be implemented by computer software executable by a data processor of a mobile device, such as a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs, also called program products, including software routines, applets and / or macros, may be stored on any device-readable data storage medium and consist of program instructions that perform certain tasks. A computer program product may consist of one or more computer-executable components configured to perform the embodiments when the program is executed. The one or more computer-executable components may be at least one software code or a portion thereof.

[0204] Further, in this regard, it should be noted that any blocks of logic flows as shown in the figures may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media, such as memory chips or memory blocks implemented within a processor, magnetic media, such as hard disks or floppy disks, and optical media, such as DVDs and their data variants, CDs. The physical media are non-transitory media.

[0205] The memory may be of any type suitable for the local technology environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed and removable memories, etc. The data processor may be of any type suitable for the local technology environment and may include, by way of non-limiting examples, one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), FPGAs, gate level circuits, and processors based on multi-core processor architectures.

[0206] The embodiments of the present disclosure can be implemented in a variety of components, such as integrated circuit modules. The design of integrated circuits is generally a highly automated process. Complex and powerful software tools are available to convert logic level designs into semiconductor circuit designs ready to be etched onto semiconductor substrates.

[0207] The scope of protection sought for various embodiments of the present disclosure is defined by the independent claims. The embodiments and features described herein (if any) that do not fall within the scope of the independent claims shall be interpreted as examples useful for understanding the various embodiments of the present disclosure.

[0208] The above description has provided a complete and informative description of the exemplary embodiments of the present disclosure by way of non-limiting examples. However, various modifications and adaptations will become apparent to those skilled in the relevant art in light of the above description, when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications of the teachings of the present disclosure will still fall within the scope of the present invention as defined by the appended claims. Indeed, there are further embodiments that consist of a combination of one or more of the embodiments with any of the other embodiments described above.

Claims

1. 1. An apparatus for rendering a communication audio signal within an immersive audio scene, the apparatus having at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured to, using the at least one processor, cause the apparatus to at least: obtaining at least one spatial audio signal for rendering within the immersive audio scene; obtaining the communication audio signal and position information associated with the communication audio signal; obtaining rendering process parameters associated with the communication audio signal and the location information; determining a rendering method based on the rendering process parameters; determining an insertion point in a rendering process for the determined rendering method and / or a selection of rendering elements for the determined rendering method based on the rendering process parameters. An apparatus configured to cause a

2. The apparatus of claim 1 , further configured to generate at least one output spatial audio signal from the at least one spatial audio signal and the communication audio signal based on the determined rendering method and the insertion point.

3. The apparatus comprises: an audio format associated with said communication audio signal; the allowable delay value, and Communication audio signal delay, The apparatus of claim 1 , adapted to determine at least one of:

4. The determined insertion point in the rendering process is then transmitted to the device. determining the insertion point in the rendering process further based on the determined at least one of an audio format, a tolerable delay value, and a communication audio signal delay associated with the communication audio signal; determining the rendering method and / or the selection of rendering elements for the determined rendering method based on the determined at least one of an audio format, a tolerable delay value, and a communication audio signal delay associated with the communication audio signal; The apparatus of claim 3 , further comprising:

5. 5. The apparatus of claim 4, wherein the allowable delay value is an amount of delay allowed for utilizing the communication audio signal, and the communication audio signal delay is a delay value determined based on an end-to-end delivery delay and a delay in rendering the communication audio.

6. The apparatus of claim 2 , wherein the generated at least one output spatial audio signal causes the apparatus to represent the communication audio signal as a high-order Ambisonic audio signal.

7. The apparatus is further adapted to obtain a user input, the apparatus being further adapted to generate the at least one output spatial audio signal based on the user input, the user input comprising: permitted communication audio signal types, Permitted audio formats, the allowable delay value, at least one acoustic modeling preference parameter; The apparatus of claim 2 , configured to define at least one of:

8. 3. The apparatus of claim 2, wherein the apparatus is further configured to obtain a communication audio signal type associated with the at least one spatial audio signal, and wherein the apparatus is further configured to generate the at least one output spatial audio signal based on the at least one communication audio signal type associated with the at least one spatial audio signal.

9. The rendering process and / or rendering element may include: Doppler processing, Direct sound processing, Material filtering, Early reflection processing, Diffuse post-reverberation processing, Sound source expansion processing, Shielding treatment, Diffraction processing, Sound source conversion processing, Externalized rendering, In-head rendering, The apparatus of claim 1 , comprising one or more of:

10. The apparatus of claim 1 , wherein the determined insertion point causes the apparatus to determine a rendering mode, the rendering mode comprising a value indicative of the insertion point of the communication audio signal.

11. The value indicating the insertion point is a first mode value indicating that the communication audio signal and the at least one spatial audio signal are inserted at the start of the rendering process; a second mode value indicating that the communication audio signal bypasses the rendering process and is mixed directly with an output of the rendering process applied to the at least one spatial audio signal; a third mode value indicating that the rendering process is fully applied to the at least one spatial audio signal, while the communication audio signal is partially rendered; The apparatus of claim 10 , comprising one of:

12. 12. The apparatus of claim 11, wherein the third mode value indicating that the communication audio signal has been partially rendered is a value indicating that the communication audio signal is a direct sound rendering relative to a point sound source and a binaural rendering relative to a user position.

13. The apparatus comprises: an audio format type of the communication audio signal based on the rendering process parameters; the insertion point in the rendering process of the communication audio signal within the rendering method determined based on the audio format type; The apparatus of claim 1 , adapted to determine at least one of:

14. 14. The apparatus of claim 13, wherein the determined insertion point in the rendering process within the determined rendering method based on the audio format type causes the apparatus to determine that if the communication audio signal has an audio format type of a pre-rendered spatial audio format, the insertion point in the rendering method is in direct mixing with an output of the rendering process applied to the at least one spatial audio signal.

15. 1. A method for an apparatus for rendering a communication audio signal in an immersive audio scene, the method comprising: obtaining at least one spatial audio signal for rendering within the immersive audio scene; obtaining the communication audio signal and position information associated with the communication audio signal; obtaining rendering process parameters associated with the communication audio signal and the position information; determining a rendering method based on the rendering process parameters; determining an insertion point in a rendering process for the determined rendering method and / or selecting a rendering element for the determined rendering method based on the rendering process parameters; The method includes:

16. The method of claim 15 , further comprising generating at least one output spatial audio signal from the at least one spatial audio signal and the communication audio signal based on the determined rendering method and insertion point.

17. an audio format associated with the communication audio signal; Tolerable delay value, Communication audio signal delay, The method of claim 15 , further comprising determining at least one of:

18. Determining the insertion point in the rendering process includes: determining the insertion point in the rendering process further based on determining at least one of the audio format associated with the communication audio signal, the allowable delay value, and the communication audio signal delay; determining the selection of rendering elements for the determined rendering method based on determining at least one of the audio format, the allowed delay value, and the communication audio signal delay associated with the rendering method and / or the communication audio signal; 20. The method of claim 17, comprising at least one of:

Images