Method and apparatus for rendering a virtual reality audio scene

Abstract

In particular, a method for rendering a virtual reality audio scene is disclosed, comprising: receiving information defining a limited area audio scene within the virtual reality audio scene (301), wherein the limited area audio scene defines a subspace of the virtual reality audio scene (304), wherein the information defines the limited area audio scene by defining a range within which a user is able to move within the virtual audio scene; determining whether a movement of the user within the limited area audio scene satisfies a condition of at least one audio scene change (302); and processing at least one audio scene change when the movement of the user within the limited area audio scene satisfies a condition of the at least one audio scene change (306).

Description

TECHNICAL FIELD

[0001] The present application relates to an apparatus and method for signaling an audio scene change with respect to an audio object within an audio scene. BACKGROUND

[0002] Virtual reality (VR) generally refers to various capturing and rendering technologies used to generate realistic images and sound that replicate a real or fictitious environment and simulate a user's physical presence in that environment. We can refer to such an environment as a content consumption space.

[0003] The way a user can move within a 3D space can be defined as a movement in six degrees of freedom. This movement is split into two categories: rotational movement and translational movement (each with three degrees of freedom). Rotational movement is sufficient to provide a simple VR experience in which the user can turn their head (pitch, yaw and roll) to experience the space from a static or automatically moving point. Translational movement means that the user can also change position within the space, thus changing their position relative to the rendering of the audio, i.e. moving along the x, y and z axes as she wishes.

[0004] Augmented reality (AR) shares many similarities with VR. Generally, it refers to a direct or indirect view of a physical reality environment that is augmented with computer-generated sensory inputs such as sound and graphics. Mixed reality (MR) is a term used to cover various levels of virtuality in the real-virtual spectrum. For most purposes, in the context of the present invention, all these terms can be used interchangeably.

[0005] The rendering of six degrees of freedom (6DoF) audio (also referred to as free viewpoint or free listening point audio) is a fast developing field. In particular, the application of audio rendering controlled by a user for AR / VR audio (e.g. the user's head rotation and position in the audio content space) is a complex and novel field. The spatial audio can for example consist of a channel-based bed and audio objects, audio objects only, or any equivalent spatial audio representation. It is understood that the audio sources themselves as well as the user can be dynamic. Thus, for example, the audio source "default" position in a 3D space can be time-varying.

[0006] Object-based audio has multiple meanings. For example, in the traditional broadcast domain, object-based audio is used to provide a wider range of individualization choices to the consumer. In this context, the typical audio element delivered as an audio object is a dialogue. This can for example allow to select a specific language version or a sports commentary associated with one of the competing teams.

[0007] With respect to 6DoF free-viewpoint audio, an audio object can be understood as a dynamic audio (source) with various properties (and very importantly, a rendering position). It is thus a definition of the audio object position or equivalent spatial audio representation that allows a user who is a listener to move around it, approach it, move away from it, and so on. Thus, the user's movements affect the way the audio is rendered. For example, some audio scenes can be complex, including a large number of audio elements represented as audio objects, where each element needs to be individually rendered spatially and temporally as the user moves around the audio scene.

[0008] However, the application of 6DoF audio rendering in AR / VR systems can be problematic due to the considerable processing overhead required to render audio scenes that include many audio elements. SUMMARY

[0009] According to a first aspect of the application, there is provided a method for rendering a virtual reality audio scene, comprising: receiving information defining a limited area audio scene within the virtual reality audio scene, wherein the limited area audio scene defines a subspace of the virtual reality audio scene, wherein the information defines the limited area audio scene by defining a range within which a user is able to move within the virtual audio scene; determining whether a movement of the user within the limited area audio scene satisfies a condition of at least one audio scene change; and processing the at least one audio scene change when the movement of the user within the limited area audio scene satisfies a condition of the at least one audio scene change.

[0010] The above-mentioned range can be defined as a number of predetermined orientation steps, wherein each of the predetermined orientation steps can have at least one associated audio scene change condition, and wherein the at least one audio scene change condition can be associated with a respective at least one audio scene change, wherein determining whether a movement of the user within the limited area audio scene satisfies a condition of at least one audio scene change can comprise determining that the movement of the user satisfies the at least one audio scene change condition of one of the number of predetermined orientation steps, and wherein processing the at least one audio scene change when the movement of the user within the limited area audio scene satisfies a condition of the at least one audio scene change can comprise processing the respective at least one audio scene change associated with the at least one audio scene change condition.

[0011] The above-mentioned range can be defined as a region of the virtual reality audio scene within a physical room, wherein the region can have at least one associated audio scene change condition, and wherein the at least one audio scene change condition can be associated with a respective at least one audio scene change, wherein determining whether the movement of the user within the limited area audio scene satisfies the condition of the at least one audio scene change can comprise determining that the movement of the user satisfies the at least one audio scene change condition of the region, and wherein processing the at least one audio scene change when the movement of the user within the limited area audio scene satisfies the condition of the at least one audio scene change can comprise processing the respective at least one audio scene change associated with the at least one audio scene change condition.

[0012] Each defined directional step or movement can be accompanied by an indication about a processing load required for processing the respective at least one audio scene change.

[0013] The information defining the limited area audio scene can be a data structure of a timed metadata track.

[0014] The above-mentioned data structure can comprise for the at least one audio scene change at least one of the following data fields: an identifier indicating an audio scene associated with a condition of the at least one audio scene change; an identifier identifying a group of audio elements comprising audio elements associated with the condition of the at least one audio scene change condition; and an identifier identifying an audio element associated with the condition of the at least one audio scene change.

[0015] The information defining the limited area audio scene can be part of a sample entry within a timed metadata track according to ISOBMFF.

[0016] The at least one audio scene change can be defined by at least one audio element and a rendering parameter of the at least one audio element.

[0017] The at least one audio element can comprise at least one of the following: an audio object; an audio channel; and a higher order ambisonics (Ambisonics).

[0018] According to a second aspect of the application, there is provided an apparatus for rendering a virtual reality audio scene, comprising: means for receiving information defining a limited area audio scene within the virtual reality audio scene, wherein the limited area audio scene defines a subspace of the virtual reality audio scene, wherein the information defines the limited area audio scene by defining a range within which a user is able to move within the virtual audio scene; means for determining whether a movement of the user within the limited area audio scene fulfils a condition of at least one audio scene change; and means for processing at least one audio scene change when the movement of the user within the limited area audio scene fulfils a condition of at least one audio scene change.

[0019] The above-mentioned range can be defined as a predetermined number of directional steps, wherein each of the predetermined directional steps can have at least one associated audio scene change condition, and wherein the at least one audio scene change condition can be associated with a respective at least one audio scene change, wherein the means for determining whether a movement of the user within the limited area audio scene fulfils a condition of at least one audio scene change can comprise means for determining that the movement of the user fulfils at least one audio scene change condition of one of the predetermined number of directional steps, and wherein the means for processing at least one audio scene change when the movement of the user within the limited area audio scene fulfils a condition of at least one audio scene change can comprise means for processing a respective at least one audio scene change associated with the at least one audio scene change condition.

[0020] The above-mentioned range can be defined as a region of the virtual reality audio scene within a physical room, wherein the region can have at least one associated audio scene change condition, and wherein the at least one audio scene change condition can be associated with a respective at least one audio scene change, wherein the means for determining whether a movement of the user within the limited area audio scene fulfils a condition of at least one audio scene change can comprise means for determining that the movement of the user fulfils at least one audio scene change condition of the region, and wherein the means for processing at least one audio scene change when the movement of the user within the limited area audio scene fulfils a condition of at least one audio scene change can comprise means for processing a respective at least one audio scene change associated with the at least one audio scene change condition.

[0021] Each defined directional step or movement can be accompanied by an indication of a processing load required for processing a respective at least one audio scene change.

[0022] The information defining the limited area audio scene can be a data structure of a timed metadata track.

[0023] The above data structure can comprise at least one of the following data fields for the at least one audio scene change: an identifier indicative of an audio scene associated with a condition of the at least one audio scene change; an identifier identifying a group of audio elements comprising audio elements associated with a condition of the at least one audio scene change condition; and an identifier identifying an audio element associated with a condition of the at least one audio scene change.

[0024] The information defining the limited area audio scene can be part of a sample entry within a timed metadata track according to ISOBMFF.

[0025] The at least one audio scene change can be defined by at least one audio element and a rendering parameter of the at least one audio element.

[0026] The at least one audio element can comprise at least one of the following: an audio object; an audio channel; and a higher order Ambisonics.

[0027] According to a third aspect of the application, there is provided an apparatus for rendering a virtual reality audio scene, comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to: receive information defining a limited area audio scene within the virtual reality audio scene, wherein the limited area audio scene defines a subspace of the virtual reality audio scene, wherein the information defines the limited area audio scene by defining a range within which a user is able to move within the virtual audio scene; determine whether a movement of the user within the limited area audio scene satisfies a condition of at least one audio scene change; and process at least one audio scene change when the movement of the user within the limited area audio scene satisfies a condition of at least one audio scene change.

[0028] A computer program product stored on a medium can cause an apparatus to perform a method as described herein.

[0029] An electronic device can comprise an apparatus as described herein.

[0030] A chipset can comprise an apparatus as described herein.

[0031] Embodiments of the present application aim to address problems associated with the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0032] For a better understanding of the present application, reference will now be made, by way of example, to the accompanying drawings in which:

[0033] Figure 1 illustrates a processing pipeline of a 6DoF audio Tenderer;

[0034] Figures 2a and 2b illustrate the extent of impact zones within a 6DoF audio scene, according to embodiments;

[0035] Figure 3 An example system adapted to implement decoding rendering and output six degrees of freedom audio, according to some embodiments, is schematically illustrated;

[0036] Figure 4 is a process flow diagram illustrating a process according to embodiments of these teachings;

[0037] Figure 5 illustrating an example of the extent of zones according to a physical room;

[0038] Figure 6 An example device adapted to implement the above illustrated apparatus, according to some embodiments, is schematically illustrated. DETAILED DESCRIPTION

[0039] Suitable apparatus and possible mechanisms for improving the decoding, rendering and presentation or output of six degrees of freedom (6DoF) audio are described in more detail below.

[0040] As previously discussed, the application of 6DoF audio rendering in AR / VR systems can be problematic due to the considerable processing overhead required in order to render an audio scene comprising many audio elements. Complex audio scenes comprising many audio elements can result in a multitude of different combinations and permutations of rendering parameters which can need to be calculated and checked in relation to the movement of a user around the audio scene. This complexity can be a considerable burden for AR / AR systems rendering 6DoF audio. One approach to reducing this computational complexity is to reduce the size of the audio scene in relation to the rendering parameters of each audio element, such that audio elements within a smaller region (around the user) are favoured over audio elements in the entire audio scene. The present invention therefore proceeds from the consideration that the complexity requirements of rendering an audio scene in relation to the movement of a user in time and space can be reduced by having an active audio scene of reduced extent.

[0041] Fig. 1 depicts a framework 6DoF AR / VR system on which embodiments can be implemented according to the Moving Picture Experts Group standard group, referred to as MPEG-I. In Fig. 1 are depicted: audio rendering data 101, which is typically streamed to a user device or stored on a user device; and an MPEG-I audio Tenderer 102, which comprises a pipeline for retrieving audio rendering data by data retrieval and pre-processing 1021 and performing auralization of rendered audio data by auralization 1022. Within the framework depicted in Fig. 1, the audio rendering data 101 can typically be considered to contain information describing a 6DoF audio scene (and more particularly, an MPEG-I 6DoF audio scene). This can encompass all content of a virtual audio scene, i.e. all sound sources (in the form of audio elements) and resource data (such as audio waveforms, source radiation patterns, and information about the acoustic environment) thereof. In addition, the audio rendering data can cater for audio scene animation by describing how the audio scene can change over time with respect to a moving object.

[0042] In the context of a 6DoF AR / VR system, and in particular in the framework of the rendering pipeline of Fig. 1, the following terms can be defined.

[0043]

[0044]

[0045] Returning to Fig. 1, it can be seen that the input 101 of the audio Tenderer 102 can comprise a plurality of audio elements, which can include any combination of audio objects, audio channels and HOA. In addition, the input 101 can also comprise audio scene information providing a description of the audio scene in the form of a file. The audio scene information file can be seen as a list of instructions that draw together various different entities such as sound sources, acoustic environments, geometric elements, audio elements into a coherent, integrated audio scene.

[0046] In some embodiments, the audio scene information can be in the form of an xml file. The following is a simple example of an xml audio scene information file, which is drawn together by a single <audioscene>Node composition, where all entities (e.g., sound sources, acoustic environment, geometric elements, audio signals, etc.) are declared. This short example defines a simple, time-invariant test scene with one MPEG-I audio object representing a trumpet with directivity.

[0047]

[0048] It will be appreciated that the above example serves only as an example of audio scene description information. Other forms of audio scene description can contain more nodes and audio elements. However, it can be seen that the audio scene information file declares the entities of the scene and contains references to data in other input files. These input files (or file pathnames) can be referenced in the declaration syntax of the entities. All pathnames are specified relative to the directory containing the scene.xml file. The audio description information file can also describe the dynamic behavior of any audio elements within the audio scene over time.

[0049] As previously discussed, the 6DoF MPEG-I audio Tenderer 102 can be considered to comprise a pipeline of data retrieval and pre-processing 1021 to retrieve audio rendering data, followed by audibleization 1022 of the rendered audio data. The retrieval of audio rendering data can be controlled in part by changes in the audio scene as a result of actions by the user. These changes can typically take the form of a list of instructions or commands that cater to various variable scenarios in the audio scene initiated by the user. For example, if the user walks into a particular zone of the audio scene, it can be required to render a particular audio element or combination of audio elements with particular rendering parameters to provide a change in the audio scene from the user's perspective. To cater to this functionality, the data retrieval and pre-processing 1021 of the 6DoF audio Tenderer 102 can include specific changes in the audio scene, in other words, an audio scene change (ASC) can be triggered by satisfying a particular ASC condition. The triggering of the ASC condition can be in response to some action initiated by the user that causes it to occur. Thus, in the 6DoF audio Tenderer 102, there can be a list of ASC conditions that trigger a change in the corresponding audio scene (ASC). In other words, for each ASC condition, there is a corresponding ASC.

[0050] The following are examples of ASC conditions:

[0051]

[0052]

[0053] In the above example, there is a proximity based defined scene change condition that enables four audio objects if the user enters a certain area in the virtual scene. Thus, it can be seen that to determine whether to trigger an ASC, multiple such ASC conditions can need to be checked. This process can occur for every frame, which can lead to an overload of the renderer.

[0054] The action of determining which ASC conditions have been met can directly influence which specific audio elements and other audio scene related information such as acoustic elements (and audio scene information) are retrieved from the audio rendering data 101 in order to construct the audio scene for the user.

[0055] Hence, a first pipeline stage of a 6DoF audio renderer can be envisaged to have a pre-processing stage, whereby each ASC condition is checked with respect to the position of the user within the audio scene. In turn, this pre-processing stage can determine which specific audio elements and other audio scene paraphernalia are retrieved from the stored / streamed audio rendering data 101.

[0056] Finally, Fig. 1 shows the last stage of the pipeline as audibilization 1022, whereby the audio elements and other audio scene related paraphernalia from the previous pipeline stage data retrieval and pre-processing 1021 are processed into the audio scene. The audibilization stage can comprise various mathematical models and signal processing functions to create the sound field of the various audio elements that make up the audio scene. Various audibilization techniques that can be used to process the rendered audio components of the audio scene are well documented in the scientific literature covering the field, to which the reader is referred.

[0057] Returning to the problem discussed above, it can be seen that if the influence range or zone of the user within the entire audio scene is limited, the subsequent processing of the audio scene around the user can be narrowed down to that range or zone. In other words, ASCs need not be processed for the entire audio scene, but only those within the influence zone of the user.

[0058] Hence, the present invention starts from the desire to define an influence zone with respect to the space of the user within the audio scene. This can have the effect of reducing the number of ASCs that are triggered due to user movement, in turn reducing the amount of processing and rendering needed to reflect changes in the audio scene.

[0059] Audio elements in a 6DoF system are rendered based on the position and rotation of the (listener) user within the virtual audio scene. Hence, depending on the movement of the user in the content consumption space (or audio scene), different audio elements become audible to the user.

[0060] With this in mind, we turn to Figures 2a and 2b, in which 6DoF audio scenes 201 and 207 are depicted, each with users 203 and 209 represented in their respective virtual audio scene. It can be seen that, in a short time period, the movement of the user can be limited to a relatively small area of the audio scene (205 and 211) compared to the total extent of the audio scene. This reduced extent of the virtual audio space can be referred to as the impact zone, since only those elements within the reduced extent are affected by the movement of the user at any time. Thus, only those audio elements associated with the reduced extent of the audio space can need to be processed in order to update the audio scene in accordance with the movement of the user. In other words, by limiting the extent of the impact zone to be less than the entire audio scene, this has the effect of limiting the number of possible ASCs that can be triggered compared to the total number of ASCs associated with the entire audio scene. This brings the advantage that a smaller number of ASC trigger conditions need to be monitored in order to cover changes to the audio scene due to movement of the user.

[0061] The size or extent of the "impact zone" can be determined in part by the range of allowable movement that the user (listener) can perform when using the application. For example, if the user 203 is in a seated position, the size of the impact zone can be determined by the possible range of movement of the head. However, with reference to Figure 2b, the user 209 is allowed to move freely within the audio scene 207. Thus, the impact zone 211 can be larger in order to accommodate a larger number of ASCs that can be triggered by movement of the user 203 within the audio scene 207.

[0062] In embodiments, the impact zone can be modelled as a virtual space around the user, and can take the form of a set such as a cube or a sphere as shown by 205 and 207 respectively in Figures 2a and 2b. Alternatively, the impact zone space can take any shape defined by a mesh consisting of a list of vertices (3D coordinates) and a number of triangular faces (i.e. indices of three vertices). The size of the possible impact zone extent can be embedded as an impact zone description timed metadata track based on the content creator's preference. In turn, the encoder can analyse the audio elements present in the audio scene to determine the bitrate impact of the impact zone size.

[0063] In some embodiments, a discrete number of impact zones can be added based on content creator preference in order to control which objects in the audio scene are accessible. This can lead to advantageous results whereby semantic coherence can be maintained. This can occur, for example, when there is a requirement to have two audio objects in the audio scene, but the restriction on the impact zone results in only one audio object being accessible. In this case, a smaller impact zone can be selected in order to maintain continuity of the audio scene.

[0064] In embodiments, the extent of the impact zone (or the bounds of the impact zone), also referred to as a finite area audio scene, can be signaled to the renderer using the following data structure.

[0065]

[0066] where distance_scale is a positive scale value indicating the unit for subsequent distance measurements in this structure. For example, a particular value of distance_scale can indicate that any distance measurement is in units of 1 cm. However, distance_scale can be defined to represent units of any suitable scale, such as meters. The variables maxX, maxY, and maxZ indicate the extent of the impact zone in units of the distance scale from the origin at the user’s head.

[0067] In some embodiments, the extent can define an area within a physical room for VR / AR audio playback, and the audio scene change would facilitate rendering the necessary audio elements / audio objects or combinations thereof to provide a VR audio scene for the defined area. Figure 5 An area 501 (defined by the extent) within a physical room 503 is depicted, where the area can be given by x and y physical dimensions.

[0068] In some embodiments, the above data structure defining the extent of the impact zone can be signaled as part of a timed metadata track. For example, the data structure defining the extent of the impact zone can be part of a sample entry within a timed metadata track encapsulated as a moov box according to the ISO Base Media File Format (ISOBMFF). In these embodiments, the sample entry of the metadata track can be used to provide continuous updates of the extent of the impact zone to the audio renderer in a timely manner, which in turn enables triggering ASC in different portions of the audio scene of an animated video. The metadata track with the impact zone sample entry can accompany any number of omnidirectional video tracks within the moov box and be applied to them.

[0069] In these embodiments, the data structure sample of the metadata track can have a track sample entry type “izod” and can be specified as:

[0070]

[0071] An example syntax for the example entry type “izod” can be specified as:

[0072]

[0073] where ZaxisSteps, XaxisSteps, and YaxisSteps are variables whose values indicate the number of steps a user can virtually move within the audio scene. Thus, since the AV / VR virtual system provides detailed information about the user's movement within the audio scene, the sample entry type "izod" provides information about which audio elements are affected by the step changes in the user's position within the audio scene. The identification of the number of audio elements affected by the user's movement, num_audio_impacted, can be indicated according to the following members of the above example data structure:

[0074] mae_audioSceneInfoID, which is an array indicating the audio scene to which the audio element affected by the step change belongs for each step;

[0075] mae_groupID, which is an array indicating the element group ID to which the audio element affected by the step change belongs for each step; and

[0076] mae_audioElementID, which is an array indicating the specific element within the group affected by the step change for each step.

[0077] Thus, each audio element can be categorized by audio scene, element group within the audio scene, and specific audio element within the element group.

[0078] As can be seen from the above structure, the ZaxisSteps, XaxisSteps, and YaxisSteps variables define the granularity of the audio elements affected by the user's movement. As such, each registered movement of the user within the sample time period will be converted into the number of steps along the three movement directions X, Y, and Z. In turn, the above "izod" sample data structure can provide an indication of which audio elements are affected for each possible range of Zaxis, Yaxis, and Zaxis steps, or pulled into each possible range of Zaxis, Yaxis, and Zaxis steps.

[0079] Additionally, each step change can also be accompanied by an indication of the bitrate requirement (or load) of the corresponding step change in the user's position, given as bitrate_impact. In other words, bitrate_impact is the bitrate impact in terms of data size or kbps, which can be provided to support a certain impact zone size, the larger the size of the step, the larger the bitrate impact.

[0080] Accordingly, the example track entry type "izod" described above allows the impact zone data to become part of a timed metadata track according to the ISOBMFF standard, thereby providing support for animated audio video scenes. Immersive content can have multiple different timed metadata tracks corresponding to different impact zone sizes. This allows the player to select a suitable impact zone according to player preferences or different impact zone description track available bandwidth influences. For example, impact zone description tracks with larger impact zone ranges are expected to have a larger bitrate impact compared to impact zone description tracks with smaller impact zone ranges.

[0081] Figure 3 An example rendering device 300, such as a 6DoF MPEG-I audio renderer, configured to implement the impact audio zone processing of the ASC described above is shown. It will be understood that the device can be implemented on any suitable hardware and software (firmware) combination. In some embodiments, the device is implemented by a digital workstation (DAW) with a VR user interface.

[0082] In addition to the streamed or stored ASC impact data, the rendering device 300 can be arranged to receive encoded audio elements and audio scene information, in other words audio rendering data according to the parlance of the pipeline in Fig. 1, from a channel or storage 301. The audio rendering data can be streamed or stored as a file format type, which can conform to ISOBMFF. In embodiments, the ASC impact data can include impact zone extent information, such as ImpactZoneExtentStruct in the case of a static audio scene, or a timed metadata track with the "izod" sample entry type described above in the case of an animated scene.

[0083] In some embodiments, there can be a specific communication channel that signals the impact zone extent information to the renderer 300.

[0084] Furthermore, in relation to the channel or storage 301, a timed metadata track can accompany the stored or streamed impact zone timed metadata track.

[0085] The renderer 300 is further arranged to receive user position and head tracker information 302 from a user 303 equipped with suitable 6DoF determination means.

[0086] Within the Tenderer 300, the Impact Zone Determiner 304 can combine the received Impact Zone range information with the user position and head tracker information in order to determine the position information of the user relative to the bounds of the Impact Zone range. In embodiments where the Impact Zone range information is deployed in the form of "izod” sample entries, this process can involve determining the closest step index along the three movement directions X, Y and Z. In turn, the output of the Impact Zone Determiner 304 can be a position within the range of the Impact Zone of the user. This information can be used to determine the particular ASCs triggered by that position.

[0087] In turn, information about the position of the user within the range of the Impact Zone can be passed to the ASC Determiner 305. The ASC Determiner 305 can be configured to use this information to determine which ASCs within the range of the Impact Zone are triggered by the position of the user. In embodiments, the determination process can take the form of determining which of the ASC conditions in a list of ASC conditions (associated with the range of the Impact Zone) are satisfied by the position of the user within the Impact Zone. As explained previously, each ASC condition can be associated with a corresponding ASC, and the ASCs can have been associated with rendering parameters of one or more audio elements within the audio scene in order to treat the audio scene as a function of the user’s perspective. It will be understood that a particular ASC can also include changes in rendering parameters of other acoustic properties of the audio scene.

[0088] In embodiments (employing the Impact Zone approach to determine ASCs), a cache processing structure can be used, in which all rendering parameters associated with the elements of each ASC are pre-computed and saved in a cache memory store. In this case, the cache processing approach can be particularly useful because only a limited number of ASCs can be triggered within the range of the Impact Zone. It is therefore feasible to pre-compute the corresponding rendering parameters and save them in the cache memory together with their ASC identification in order to improve processing efficiency.

[0089] In one operational scenario for the Tenderer 300, the Tenderer 300 can receive the Impact Zone range information via the application interface. In turn, the Tenderer 300 can check the ASC conditions within the Impact Zone as determined by the position and orientation of the user. The ASC conditions can be pre-stored in a cache. If any particular ASC condition is triggered by the data resulting from the position and orientation of the user, the corresponding ASC and accompanying audio elements (including the corresponding rendering parameters) can be retrieved for audibleization of the audio scene.

[0090] In another operational scenario of the Tenderer 300, as previously mentioned, the Tenderer 300 can receive impact zone extent information via the application interface. In addition, the Tenderer 300 can receive a plurality of timed metadata tracks, each carrying a different audio element related to a different impact zone extent area (impact zone description timed metadata track). In turn, the Tenderer can select the appropriate impact zone description timed metadata track for a particular impact zone. In turn, the Tenderer 300 can determine which ASC conditions are triggered from the selected impact zone description timed metadata track. The impact zone description timed metadata track can have been incorporated within the bitrate_impact data structure, allowing the Tenderer to determine the processing requirements for the selected timed metadata track.

[0091] Accordingly, the output of the ASC determiner 305 can include audio rendering parameters associated with each triggered ASC within the impact zone extent. These parameters can be passed from the channel and storage 301 to the audibilization implementation 306 along with the audio elements and acoustic environment metadata. In turn, the audibilization implementation 306 can process the audio elements and acoustic characteristics associated with the triggered ASCs in order to produce a virtual audio scene. As previously mentioned, the audibilization implementation 306 can deploy a variety of techniques to implement this, such as decoding and signal processing techniques for processing the audio elements and various mathematical models for processing the acoustic room features in order to create the sound field that constitutes the audio scene. The output 307 of the audibilization implementation 306 (i.e., the rendered audio signal) can be presented to the user 303, for example, through headphones.

[0092] Figure 4 A set of processing steps that can be processed by a 6DoF audio Tenderer in order to render ASCs within an impact zone is shown. With reference to Figure 3 The following processing steps can be performed in the scope of the impact zone determiner 304 and the ASC determiner 305.

[0093] The processing steps of the 6DoF audio Tenderer can be configured to receive impact zone extent information and user position and orientation information, which is described in more detail in Figure 4 The two pieces of information are shown as processing steps 401 and 403. As mentioned above, these pieces of information can be used to determine the position of the user relative to the impact zone. In this regard, processing step 405 is described as using the two pieces of information described above to determine the position of the user in the impact zone. The result of this processing step is used to determine which ASC conditions are triggered within the impact zone. This is shown as processing step 407, where it can be seen that the ASC conditions associated with the impact zone are received at processing step 409. In turn, the output of 407 is passed to decision processing step 411, which determines whether the ASC id is present in the cache memory. The cache can comprise at least one of: the ASC condition ID; the ASC condition; and the ASC condition and associated rendering parameters.

[0094] If the ASC has not already been added to the cache, the processed audio rendering parameters of the audio element associated with the triggered ASC are added to the cache with the ASC id. This is shown as processing step 413. Figure 4 Finally, as shown in processing step 415 in Figure 4 , the processed audio rendering parameters of the audio element associated with the triggered ASC are shown as the output of the cache.

[0095] With regard to Figure 6 , an example electronic device that can be used as a capture device and / or an audio signal analyzer / processor and / or a playback device is shown. The device can be any suitable electronic device or apparatus. For example, in some embodiments, the device 1400 is a mobile device, user equipment, tablet, computer, audio playback apparatus, etc.

[0096] The device 1400 can comprise a microphone or microphone array 1401. The microphone or microphone array 1401 can comprise a plurality (e.g. N) of microphone units. However, it will be appreciated that there can be any suitable microphone configuration and any suitable number of microphones. In some embodiments, the microphone or microphone array 1401 is separate from the apparatus and the audio signals are sent to the apparatus by a wired or wireless coupling. In some embodiments, the microphone or microphone array 1401 can be a microphone array as shown in the previous figures.

[0097] The microphone or microphone array can include a transducer configured to convert sound waves into suitable electrical audio signals. In some embodiments, the microphone or microphone array can include a solid-state microphone. In other words, the microphone can be capable of capturing audio signals and outputting suitable digital format signals. In some other embodiments, the microphone or microphone array 1401 can include any suitable microphone type or audio capturing component, for example, a condenser microphone, a capacitor microphone, an electrostatic microphone, an electret condenser microphone, a dynamic microphone, a ribbon microphone, a carbon microphone, a piezoelectric microphone, or a microelectromechanical systems microphone. In some embodiments, the microphone or microphone array can output the captured audio signals to an analog-to-digital converter (ADC) 1403.

[0098] The device 1400 can further include an analog-to-digital converter 1403. The analog-to-digital converter 1403 can be configured to receive audio signals from each microphone 1401 and convert them into a format suitable for processing. In some embodiments where the microphone or microphone array includes an integrated microphone, an analog-to-digital converter is not needed. The analog-to-digital converter 1403 can be any suitable analog-to-digital conversion or processing component. The analog-to-digital converter 1403 can be configured to output digital representations of the audio signals to the processor 1407 or the memory 1411.

[0099] In some embodiments, the device 1400 includes at least one processor or central processing unit 1407. The processor 1407 can be configured to execute various program codes, such as the methods described herein.

[0100] In some embodiments, the device 1400 includes a memory 1411. In some embodiments, the at least one processor 1407 is coupled to the memory 1411. The memory 1411 can be any suitable storage component. In some embodiments, the memory 1411 includes a program code portion for storing program codes that can be implemented on the processor 1407. Further, in some embodiments, the memory 1411 can further include a storage data portion for storing data, for example, data that has been processed or is to be processed according to the embodiments described herein. The processor 1407 can retrieve implemented program codes stored in the program code portion and data stored in the storage data portion via the memory-processor coupling, as needed.

[0101] In some embodiments, the device 1400 includes a user interface 1405. In some embodiments, the user interface 1405 can be coupled to the processor 1407. In some embodiments, the processor 1407 can control operation of the user interface 1405 and receive input from the user interface 1405. In some embodiments, the user interface 1405 can enable a user to input commands, e.g., via a keyboard, to the device 1400. In some embodiments, the user interface 1405 can enable a user to obtain information from the device 1400. For example, the user interface 1405 can include a display configured to display information from the device 1400 to a user. In some embodiments, the user interface 1405 can include a touch screen or touch interface that can enable information to be input to the device 1400 and also display information to a user of the device 1400. In some embodiments, the user interface 1405 can be a user interface for communicating with a position determiner as described herein.

[0102] In some embodiments, the device 1400 includes a transceiver 1409. In such embodiments, the transceiver 1409 can be coupled to the processor 1407 and configured to enable communication with other apparatuses or electronic devices, e.g., via a wireless communication network. In some embodiments, the transceiver 1409 or any suitable transceiver or transmitter and / or receiver apparatus can be configured to communicate with other electronic devices or apparatuses via a wired or wired coupling.

[0103] For example, the transceiver 1409 can be configured to communicate with a renderer as described herein.

[0104] The transceiver 1409 can communicate with other apparatuses by any suitable known communication protocol. For example, in some embodiments, the transceiver 1409 or transceiver components can use suitable Universal Mobile Telecommunications System (UMTS) protocols, wireless local area network (WLAN) protocols such as, e.g., IEEE 802.X, suitable short-range radio frequency communication protocols such as Bluetooth, or infrared data communication paths (IRDA).

[0105] In some embodiments, the device 1400 can be used as at least part of an audio processor. Thus, the transceiver 1409 can be configured to receive audio signals and position information from a capture device microphone or microphone array, and in some embodiments, determine parameters as described herein by execution of suitable code using the processor 1407. Further, the device can generate suitable audio signals and parameter outputs to be transmitted to a renderer or spatial processing device.

[0106] In some embodiments, device 1400 can be used as at least part of a renderer. Thus, transceiver 1409 can be configured to receive audio signals from a microphone or microphone array, and in some embodiments, receive parameters determined at a capture device or processing device as described herein, and generate a suitable audio signal rendering by execution of suitable code using processor 1407. Device 1400 can include digital-to-analog converter 1413. Digital-to-analog converter 1413 can be coupled to processor 1407 and / or memory 1411 and configured to convert a digital representation of an audio signal, such as from processor 1407 after audio rendering of the audio signal as described herein, into a suitable analog format suitable for presentation via an audio subsystem output. In some embodiments, digital-to-analog converter (DAC) 1413 or signal processing components can be any suitable DAC technology.

[0107] Further, in some embodiments, device 1400 can include audio subsystem output 1415. The example shown in FIG. 14A shows audio subsystem output 1415 as an output jack configured to enable coupling with earphones 1421. However, audio subsystem output 1415 can be any suitable audio output or connection to an audio output. For example, audio subsystem output 1415 can be a connection to a multi-channel speaker system. Figure 6

[0108] In some embodiments, digital-to-analog converter 1413 and audio subsystem output 1415 can be implemented within physically separate output devices. For example, DAC 1413 and audio subsystem output 1415 can be implemented as a wireless earphone in communication with device 1400 via transceiver 1409.

[0109] While device 1400 is shown with audio capture, audio processing, and audio rendering components, it will be understood that, in some embodiments, device 1400 can include only portions of these elements.

[0110] In general, the various embodiments of the application can be implemented in hardware or special-purpose circuits, software, logic or any combination thereof. For example, some aspects can be implemented in hardware, while other aspects can be implemented in

[0111] ​Embodiments of the application can be implemented by computer software executable by a data processor of the mobile device such as in the processor entity, or by hardware, or by a combination of software and hardware. Further, in this regard, it should be noted that any blocks of the logic flow of the accompanying figures can represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software can be stored on such physical media as memory chips, or memory blocks implemented in the processor, magnetic media such as hard disk or floppy disks, and optical media such as for example DVD and the data variants thereof CD. The software can also be stored in memory of another computer or device that is able to transmit it over a wired or wireless interface to the mobile device.

[0112] The memory can be of any type appropriate for the local technical environment and can be implemented using any appropriate data storage technology, such as semiconductor based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The data processor can be of any type appropriate for the local technical environment, and can include one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASIC), gate level circuits and processors based on multi core processor architectures, as non limiting examples.

[0113] Embodiments of the application can be practiced in a variety of components such as integrated circuit modules. The design of integrated circuits is by nature highly automated. Complex and powerful software programs are available for design, simulation, test, and manufacture of the circuits. In this manner, the practical implementation of the application can be achieved.

[0114] The foregoing description has provided by way of exemplary and non-limiting examples a full and informative description of exemplary embodiments of the application. However, various modifications and adaptations taken and are aspects of the above described embodiments fall within the scope of the present application as defined by the appended claims. However, all such and similar modifications and adaptations as fall within the scope of the teachings of the present application are intended to fall within the scope of the present application as defined by the appended claims.< / audioscene>

Claims

1. A method for rendering a virtual reality audio scene, comprising: receiving information defining a limited area audio scene within the virtual reality audio scene, wherein the limited area audio scene defines a subspace of the virtual reality audio scene, wherein the information defines the limited area audio scene by defining a range within which a user is able to move within the virtual reality audio scene, wherein the range is defined as a number of predetermined orientation steps, wherein each of the predetermined orientation steps has at least one associated audio scene change condition, wherein the at least one audio scene change condition is associated with a respective at least one audio scene change, and wherein each predetermined orientation step is accompanied by an indication of a bitrate requirement for processing the respective at least one audio scene change; determining whether the movement of the user within the limited area audio scene satisfies a condition of at least one audio scene change by determining that the movement of the user satisfies the at least one audio scene change condition of one of the number of predetermined orientation steps; and processing the at least one audio scene change when the movement of the user within the limited area audio scene satisfies the condition of the at least one audio scene change.

2. The method for rendering a virtual reality audio scene of claim 1, wherein, the range is defined as an area of the virtual reality audio scene within a physical room, wherein the area has at least one associated audio scene change condition, and wherein the at least one audio scene change condition is associated with a respective at least one audio scene change, wherein determining whether the movement of the user within the limited area audio scene satisfies a condition of at least one audio scene change comprises determining that the movement of the user satisfies the at least one audio scene change condition of the area, and wherein processing the at least one audio scene change when the movement of the user within the limited area audio scene satisfies the condition of the at least one audio scene change comprises processing the respective at least one audio scene change associated with the at least one audio scene change condition.

3. The method for rendering a virtual reality audio scene according to claim 1 or 2, wherein, the information defining the limited area audio scene is a data structure of a timed metadata track.

4. The method for rendering a virtual reality audio scene of claim 3, wherein, the data structure comprises at least one of at least the following data fields for the at least one audio scene change: an identifier indicating an audio scene associated with the condition of the at least one audio scene change; an identifier identifying a group of audio elements comprising audio elements associated with the condition of the at least one audio scene change condition; and an identifier identifying an audio element associated with the condition of the at least one audio scene change.

5. The method for rendering a virtual reality audio scene of claim 3, wherein, the information defining a limited area audio scene is part of a sample entry within a timed metadata track according to ISOBMFF.

6. The method of rendering a virtual reality audio scene of claim 1, wherein, the at least one audio scene change is defined by at least one audio element and rendering parameters of the at least one audio element.

7. The method for rendering a virtual reality audio scene according to claim 6, wherein, the at least one audio element comprises at least one of: an audio object; an audio channel; and a higher order ambisonic.

8. An apparatus for rendering a virtual reality audio scene, comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to: receiving information defining a limited area audio scene within the virtual reality audio scene, the limited area audio scene defining a subspace of the virtual reality audio scene, wherein, define the limited area audio scene by information defining a range within which a user is able to move within the virtual reality audio scene, wherein the range is defined as a number of predetermined orientation steps, wherein each of the predetermined orientation steps has at least one associated audio scene change condition, wherein the at least one audio scene change condition is associated with a respective at least one audio scene change, and wherein each predetermined orientation step is accompanied by an indication of a bitrate requirement to handle the respective at least one audio scene change; determine whether the movement of the user within the limited area audio scene fulfills a condition of at least one audio scene change by determining that the movement of the user fulfills at least one audio scene change condition of one of the number of predetermined orientation steps; and handle the at least one audio scene change when the movement of the user within the limited area audio scene fulfills the condition of the at least one audio scene change.

9. The apparatus for rendering a virtual reality audio scene of claim 8, wherein, the range is defined as an area of the virtual reality audio scene within a physical room, wherein the area has at least one associated audio scene change condition, and wherein the at least one audio scene change condition is associated with a respective at least one audio scene change, wherein the apparatus caused to determine whether the movement of the user within the limited area audio scene fulfills a condition of at least one audio scene change is caused to determine that the movement of the user fulfills the at least one audio scene change condition of the area, and wherein the apparatus caused to handle the at least one audio scene change when the movement of the user within the limited area audio scene fulfills the condition of the at least one audio scene change is caused to handle the respective at least one audio scene change associated with the at least one audio scene change condition.

10. The apparatus for rendering a virtual reality audio scene according to claim 8 or 9, wherein, the information defining the limited area audio scene is a data structure of a timed metadata track.

11. The apparatus for rendering a virtual reality audio scene of claim 10, wherein, the data structure comprises for the at least one audio scene change at least one of the following data fields: an identifier indicating an audio scene associated with the condition of the at least one audio scene change; an identifier identifying a group of audio elements comprising audio elements associated with the condition of the at least one audio scene change condition; and an identifier identifying an audio element associated with the condition of the at least one audio scene change.

12. The apparatus for rendering a virtual reality audio scene of claim 10, wherein, the information defining a limited area audio scene is part of a sample entry within a timed metadata track according to ISOBMFF.

13. The apparatus for rendering a virtual reality audio scene of claim 8, wherein, the at least one audio scene change is defined by at least one audio element and rendering parameters of the at least one audio element.

14. The apparatus for rendering a virtual reality audio scene according to claim 13, wherein, the at least one audio element comprises at least one of: an audio object; an audio channel; And Higher order ambisonics.

Images