Method and apparatus for audio transitions between acoustic environments

Abstract

The present invention provides a method and apparatus for enabling audio transitions between acoustic environments (AEs). [Solution] A method for enabling audio transitions between at least two AEs includes the step of obtaining at least a first AE information associated with an audio scene. The audio scene comprises a first AE and a second AE. The method also includes the step of obtaining a first distance threshold that at least partially defines an audio transition region that enables adaptive rendering between the first AE and the second AE, depending on the listening position within the audio scene; the step of determining the listening position to adjust the environmental characteristics of at least one of the first AE and the second AE; and the step of adjusting the environmental characteristics according to the listening position.

Description

Technical Field

[0001] This application relates to a method and apparatus for implementing a delay reverberation crossfade between acoustic environments in an immersive audio scene, and more particularly to a method and apparatus for implementing a delay reverberation crossfade between acoustic environments in an immersive audio scene for six-degree-of-freedom rendering, but is not limited thereto.

Background Art

[0002] Augmented reality (AR) applications (and other similar virtual scene creation applications such as mixed reality (MR) and virtual reality (VR)) that present a virtual scene to a user wearing a head-mounted device (HMD) have become more complex and sophisticated over time. The applications can include data that includes visual components (or overlays) and audio components (or overlays) presented to the user. These components can be provided to the user in accordance with the position and orientation of the user (for six-degree-of-freedom applications) within the augmented reality (AR) scene.

[0003] Scene information for rendering an AR scene typically consists of two parts. One part is virtual scene information, which may be described during content creation (or by a suitable capture device or apparatus) and represents the captured (or initially generated) scene. The virtual scene may be provided in an encoder input format (EIF) data format. The EIF and (captured or generated) audio data are used by the encoder to generate the scene description and spatial audio metadata (and encoded audio signal), which may be delivered via a bitstream to the rendering (playback) device or apparatus. EIF is described in the MPEG-I 6DoF audio encoder input format developed for the ISO / IEC JTC1 SC29 WG6 Proposal for MPEG-I 6DoF audio encoding (CfP). Implementation forms are primarily described herein, but other scene description formats that may be provided or used by the scene / content creator may also be used.

[0004] According to EIF, encoder input data contains information describing an MPEG-I 6DoF audio scene. This covers all the contents of the virtual auditory scene, i.e., all its sound sources, and resource data such as audio waveforms, sound source emission patterns, and information about the acoustic environment. Thus, the content can include both audio-generating elements such as objects, channels, and higher-order ambisonics, and their metadata (such as position and orientation and source directivity patterns), and non-audio-generating elements (such as acoustically relevant scene geometry and material properties). The input data also allows describing changes in the scene. These changes, called updates, can occur at different times and can animate the scene (e.g., moving objects). Alternatively, they can be updated manually, triggered by conditions (e.g., a listener enters proximity), or dynamically by an external entity.

[0005] Therefore, EIF information defines the created or captured acoustic environment. This can be modeled in some situations as a series of acoustically coupled (i.e., connected to allow audio transmission) acoustic environments. Thus, in addition to the acoustic environment for specifying reverberation characteristics, the acoustically coupled space requires the concept of "portals" to make the overall rendering realistic. Acoustic portals have the ability to render reverberation from adjacent rooms. Portals can be modeled or rendered as sound sources, particularly in acoustic environments that have audio supplied from the reverberation of adjacent acoustic environments. [Overview of the project]

[0006] An apparatus for enabling audio transitions between at least two acoustic environments, the apparatus comprising means configured to perform: acquiring information of at least a first acoustic environment associated with an audio scene, wherein the audio scene comprises the first acoustic environment and the second acoustic environment; acquiring a first distance threshold that at least partially defines an audio transition region that enables adaptive rendering between the first and second acoustic environments depending on the listening position within the audio scene; determining the listening position and adjusting the environmental characteristics of at least one of the first and second acoustic environments; and adjusting the environmental characteristics of at least one of the first and second acoustic environments depending on the listening position, wherein the environmental characteristics are adaptively controlled within the audio scene.

[0007] A means configured to acquire information of at least a first acoustic environment associated with an audio scene, wherein the audio scene comprises a first acoustic environment and a second acoustic environment, and the means is configured to receive a bitstream containing information of at least a first acoustic environment associated with the audio scene.

[0008] The bitstream may further comprise a first distance threshold, and a means configured to acquire the first distance threshold may be configured to acquire the first distance threshold from the bitstream.

[0009] The first acoustic environment and the second acoustic environment can be coupled by a first acoustic coupling located at the boundary between the first acoustic environment and the second acoustic environment.

[0010] A first distance threshold that at least partially defines the audio transition region is located within the first acoustic environment and can be associated with the second acoustic environment.

[0011] Means configured to adjust the characteristics of at least one of the first and second acoustic environments depending on the listening position may be configured to determine that the listener position is within an audio transition region and to adjust at least one of the first reverberation gain parameter of the first acoustic environment or the second reverberation gain parameter of the second acoustic environment based on the listening position relative to the boundary between the first and second acoustic environments.

[0012] The means may be further configured to acquire a first function related to a first distance threshold, and the means configured to adjust the environmental characteristics of at least one of the first and second acoustic environments depending on the listening position may be configured to adjust at least one of a first reverberation gain parameter of the first acoustic environment or a second reverberation gain parameter of the second acoustic environment, based on the first function applied to the listening position with respect to the first distance threshold and the boundary between the first and second acoustic environments.

[0013] The means of the present invention can be further configured to render a spatial audio signal, which can be at least partially composed of reverberation generated based on the characteristics of at least one of the first and second acoustic environments.

[0014] Means configured to render a spatial audio signal can be configured to generate a spatial audio signal having a first reverberation portion based on a first reverberation gain parameter of a first acoustic environment, and / or to generate a second reverberation portion based on a second reverberation gain parameter of a second acoustic environment.

[0015] Means configured to generate a spatial audio signal having a first reverberation gain parameter based on a second reverberation gain parameter, which is based on a first reverberation gain parameter of a first acoustic environment and / or a second reverberation gain parameter of a second acoustic environment, can be configured to perform the steps of: setting the current acoustic environment as the first acoustic environment when the listener position is within the first acoustic environment; setting the reverberation decay of all other acoustic environments separately from the first acoustic environment so as not to provide a contribution; setting the reverberation decay to provide a defined contribution to the second acoustic environment, wherein the listener position is within a region of the first acoustic environment defined by a threshold; determining the reverberation contributions of the current acoustic environment and the other acoustic environments; and combining the reverberation contributions to form part of a spatial audio signal.

[0016] Means configured to set reverberation decay so as to provide a defined contribution for a second acoustic environment where the listener position is within a region of a first acoustic environment defined by a threshold can be configured to set reverberation decay so as to provide a defined contribution for the second acoustic environment based on the first function.

[0017] Means configured to acquire a first distance threshold that at least partially defines a first distance threshold that enables adaptive rendering between first and second acoustic environments depending on the listener position in an audio scene may be configured to acquire at least one of the following: information indicating the number of acoustic environments to which the first distance threshold is applicable; information indicating the acoustic environments to which the first distance threshold is applicable; information indicating which acoustic environment the first distance threshold is located in; information indicating a distance value relative to the first distance threshold; information indicating a shape or profile associated with the first distance threshold; information indicating a hysteresis offset value that defines a hysteresis region; information indicating an attenuation modulation type that defines a function for correcting the attenuation level when the listener position moves from outside the first distance threshold to the boundary between the first and second acoustic environments; information indicating hops that define the number of hops from the current acoustic environment to the desired acoustic environment; and information indicating a parameter attenuation upper limit that defines the maximum attenuation level.

[0018] According to a second aspect, an apparatus is provided for generating acoustic environment information to assist in rendering an audio scene, the apparatus comprising means configured to acquire information of at least a first acoustic environment related to an audio scene, wherein the audio scene comprises a first acoustic environment and a second environment; acquire a first distance threshold that at least partially defines an audio transition region that enables adaptive rendering between the first and second acoustic environments depending on the listening position in the audio scene; and encode and output the information and the first distance threshold.

[0019] The means of the present invention can be further configured to acquire a first function associated with a first distance threshold, the first function defining an environmental modification characteristic of at least one of a first acoustic environment and a second acoustic environment.

[0020] Means configured to acquire a first distance threshold that at least partially defines a first distance threshold that enables adaptive rendering between first and second acoustic environments depending on the listener position in an audio scene may be configured to acquire at least one of the following: information indicating the number of acoustic environments to which the first distance threshold is applicable; information indicating the acoustic environments to which the first distance threshold is applicable; information indicating which acoustic environment the first distance threshold is located; information indicating a distance value relative to the first distance threshold; information indicating a shape or profile associated with the first distance threshold; information indicating a hysteresis offset value that defines a hysteresis region; information indicating an attenuation modulation type that defines a function for correcting the attenuation level when the listener position moves from outside the first distance threshold to the boundary between the first and second acoustic environments; information indicating hops that define the number of hops from the current acoustic environment to the target acoustic environment; and information indicating a parameter attenuation upper limit that defines the maximum attenuation level.

[0021] A means configured to encode and output information may be configured to generate a bitstream comprising encoded information and a first distance threshold. The means may be further configured to acquire at least one audio signal associated with an audio scene, and the means configured to encode and output information may be configured to generate a bitstream comprising at least one encoded audio signal.

[0022] According to a third aspect, an apparatus is provided for enabling audio transitions between at least two acoustic environments, the method comprising: acquiring at least first acoustic environment information relating to an audio scene, wherein the audio scene comprises a first acoustic environment and a second acoustic environment; acquiring a first distance threshold that at least partially defines an audio transition region enabling adaptive rendering between the first acoustic environment and the second acoustic environment, depending on the listener position in the audio scene; determining the listener position in order to adjust the environmental characteristics of at least one of the first acoustic environment and the second acoustic environment; and adjusting the environmental characteristics of at least one of the first acoustic environment and the second acoustic environment, depending on the listener position, wherein the environmental characteristics are adaptively controlled within the audio scene.

[0023] Obtaining information about at least a first acoustic environment associated with an audio scene means that the audio scene includes a first acoustic environment, and the second acoustic environment may include information about at least a first acoustic environment associated with the audio scene, and hereby includes receiving a bitstream that includes information about at least a first acoustic environment associated with the audio scene.

[0024] A bitstream may further have a first distance threshold, and obtaining the first distance threshold may include obtaining the first distance threshold from the bitstream.

[0025] The first acoustic environment and the second acoustic environment can be coupled by a first acoustic coupling located at the boundary between the first acoustic environment and the second acoustic environment.

[0026] The first distance threshold that at least partially defines the audio transition region is located within the first acoustic environment and can be associated with the second acoustic environment.

[0027] The step of adjusting at least one environmental characteristic of the first and second acoustic environments according to the listening position includes determining that the listener position is within the audio transition region, and based on the first distance threshold and the listener position relative to the boundary between the first acoustic environment and the second acoustic environment, adjusting at least one of the first reverberation gain parameters of the first acoustic environment or the second reverberation gain parameters of the second acoustic environment.

[0028] The method can further include obtaining a first function related to the first distance threshold, and adjusting at least one environmental characteristic of the first and second acoustic environments according to the listening position can further include adjusting at least one of the first reverberation gain parameters of the first acoustic environment or the second reverberation gain parameters of the second acoustic environment based on the first function applied to the listening position with respect to the first distance threshold and the boundary between the first acoustic environment and the second acoustic environment.

[0029] The method can further include rendering a spatial audio signal, and the spatial audio signal can at least partially include reverberation generated based on at least one environmental characteristic of the first and second acoustic environments.

[0030] Rendering the spatial audio signal can include generating a spatial audio signal having a first reverberation portion based on the first reverberation gain parameter of the first acoustic environment, and / or generating a second reverberation portion based on the second reverberation gain parameter of the second acoustic environment.

[0031] Generating a spatial audio signal having a first reverberation gain parameter based on the first and / or second reverberation gain parameters of the first acoustic environment, based on the second reverberation gain parameter of the second acoustic environment, may include setting the current acoustic environment as the first acoustic environment when the listener position is within the first acoustic environment; setting reverberation attenuation for all other acoustic environments separately from the first acoustic environment so as not to provide a contribution; setting reverberation attenuation so as to provide a defined contribution for the second acoustic environment when the listener position is within a region of the first acoustic environment defined by a threshold; determining the reverberation contributions for the current acoustic environment and other acoustic environments; and combining the reverberation contributions to form part of the spatial audio signal.

[0032] Setting reverberation decay to provide a defined contribution for a second acoustic environment where the listener position is within a region of a first acoustic environment defined by a threshold may include setting reverberation decay to provide a defined contribution for a second acoustic environment based on a first function.

[0033] Obtaining a first distance threshold that at least partially defines an audio transition region enabling adaptive rendering between first and second acoustic environments depending on the listener's position within an audio scene may include obtaining at least one of the following: information indicating the number of acoustic environments to which the first distance threshold is applicable; information indicating the acoustic environments to which the first distance threshold is applicable; information indicating which acoustic environment the first distance threshold is located; information indicating the distance value relative to the first distance threshold; information indicating the shape or profile associated with the first distance threshold; information indicating a hysteresis offset value defining a hysteresis region; information indicating an attenuation modulation type defining a function for correcting the attenuation level when the listener's position moves from outside the first distance threshold to the boundary between the first and second acoustic environments; information indicating hops defining the number of hops from the current acoustic environment to the desired acoustic environment; and information indicating a parameter attenuation upper limit defining the maximum attenuation level.

[0034] According to a fourth aspect, a method is provided for an apparatus for generating acoustic environment information to assist in rendering an audio scene, the method comprising: acquiring at least first acoustic environment information relating to an audio scene, wherein the audio scene comprises a first acoustic environment and a second environment; acquiring a first distance threshold that at least partially defines an audio transition region enabling adaptive rendering between the first acoustic environment and the second acoustic environment depending on the listening position in the audio scene; and encoding and outputting the information and the first distance threshold.

[0035] The method may further include obtaining a first function related to a first distance threshold, the first function defining the tuning of at least one environmental characteristic of a first and second acoustic environment.

[0036] Obtaining a first distance threshold that at least partially defines an audio transition region enabling adaptive rendering between first and second acoustic environments depending on the listener's position within an audio scene may include obtaining at least one of the following: information indicating the number of acoustic environments to which the first distance threshold is applicable; information indicating the acoustic environments to which the first distance threshold is applicable; information indicating which acoustic environment the first distance threshold is located; information indicating the distance value relative to the first distance threshold; information indicating the shape or profile associated with the first distance threshold; information indicating a hysteresis offset value defining a hysteresis region; information indicating an attenuation modulation type defining a function for correcting the attenuation level when the listener's position moves from outside the first distance threshold to the boundary between the first and second acoustic environments; information indicating hops defining the number of hops from the current acoustic environment to the desired acoustic environment; and information indicating a parameter attenuation upper limit defining the maximum attenuation level.

[0037] Encoding and outputting information may involve generating a bitstream comprising the encoded information and a first distance threshold.

[0038] The method of the present invention may further include the step of acquiring at least one audio signal associated with an audio scene, and the step of encoding and outputting information may include the step of generating a bitstream having the encoded at least one audio signal.

[0039] According to a fifth aspect, there is a device comprising at least one memory containing computer program code, wherein the at least one memory and the computer program code are configured to use the at least one processor to cause the device to acquire information of at least one first acoustic environment relating to an audio scene, where the audio scene comprises a first acoustic environment and a second acoustic environment, and the device acquires a first distance threshold that at least partially defines an audio transition region that enables adaptive rendering between the first acoustic environment and the second acoustic environment depending on the listener position in the audio scene, determines a listening position to adjust the environmental characteristics of at least one of the first and second acoustic environments, adjusts the environmental characteristics of at least one of the first and second acoustic environments depending on the listener position, and the environmental characteristics are adaptively controlled within the audio scene.

[0040] A device that acquires information about at least the first acoustic environment related to an audio scene having a first acoustic environment can receive a bitstream in which the second acoustic environment has information about at least the first acoustic environment related to the audio scene.

[0041] The bitstream may further comprise a first distance threshold, and a means configured to acquire the first distance threshold may be configured to acquire the first distance threshold from the bitstream.

[0042] The first acoustic environment and the second acoustic environment can be coupled by a first acoustic coupling located at the boundary between the first acoustic environment and the second acoustic environment.

[0043] A first distance threshold that at least partially defines the audio transition region is located within the first acoustic environment and can be associated with the second acoustic environment.

[0044] A device that adjusts the characteristics of at least one of the first and second acoustic environments according to the listening position can determine that the listener position is within an audio transition region and adjust at least one of the first reverberation gain parameter of the first acoustic environment or the second reverberation gain parameter of the second acoustic environment based on a first distance threshold and the listening position relative to the boundary between the first and second acoustic environments.

[0045] The device can be made to acquire a first function related to a first distance threshold, and to adjust the environmental characteristics of at least one of the first and second acoustic environments depending on the listening position. The device can be made to adjust at least one of the first reverberation gain parameter of the first acoustic environment or the second reverberation gain parameter of the second acoustic environment, based on the first function applied to the listening position relative to the first distance threshold and the boundary between the first and second acoustic environments.

[0046] The apparatus of the present invention can be configured to render a spatial audio signal, which may comprise at least partially reverberation generated based on the characteristics of at least one of the first and second acoustic environments.

[0047] A device for rendering spatial audio signals can generate a spatial audio signal having a first reverberation portion based on a first reverberation gain parameter of a first acoustic environment, and / or generate a second reverberation portion based on a second reverberation gain parameter of a second acoustic environment.

[0048] A device that causes a spatial audio signal to be generated including a first reverberation portion based on a first reverberation gain parameter of a first acoustic environment, and / or causes a spatial audio signal to be generated including a second reverberation portion based on a second reverberation gain parameter, can perform the following actions when the listener position is within the first acoustic environment: setting the current acoustic environment as the first acoustic environment; setting reverberation attenuation for all other acoustic environments separately from the first acoustic environment so as not to provide a contribution; setting reverberation attenuation so as to provide a defined contribution for the second acoustic environment which is within a region of the first acoustic environment defined by a threshold; determining the reverberation contributions for the current acoustic environment and other acoustic environments; and combining the reverberation contributions to form part of a spatial audio signal.

[0049] A device that sets reverberation attenuation to provide a defined contribution for a second acoustic environment where the listener position is within a region of a first acoustic environment defined by a threshold can be configured to set reverberation attenuation to provide a defined contribution for the second acoustic environment based on a first function.

[0050] A device that acquires a first distance threshold that at least partially defines an audio transition region that enables adaptive rendering between a first acoustic environment and a second acoustic environment depending on the listener's position in an audio scene can acquire at least one of the following: information indicating the number of acoustic environments to which the first distance threshold is applicable; information indicating the acoustic environments to which the first distance threshold is applicable; information indicating which acoustic environment the first distance threshold is located in; information indicating the distance value for the first distance threshold; information indicating the shape or parameters associated with the first distance threshold; information indicating a hysteresis offset value that defines a hysteresis region; information indicating an attenuation modulation type that defines a function to change the attenuation level, wherein when the listener's position moves from outside the first distance threshold to the boundary between the first acoustic environment and the second acoustic environment, information indicating an attenuation modulation type that defines a function to change the attenuation level; hop information indicating the number of hops from the current sound environment to the destination sound environment; and information indicating a parameter attenuation limit value that indicates the maximum amount of attenuation.

[0051] According to the sixth aspect, an apparatus is provided comprising at least one processor and at least one storage device including computer program code, wherein the at least one storage device and the computer program code are configured to cause the apparatus to use at least one processor to acquire information of at least a first acoustic environment related to an audio scene, the audio scene comprising a first acoustic environment and a second environment, to acquire a first distance threshold that at least partially defines an audio transition region that enables adaptive rendering between the first acoustic environment and the second acoustic environment depending on the listener position in the audio scene, and to encode and output the information and the first distance threshold.

[0052] The device can further be made to acquire a first function related to a first distance threshold, the first function defining the tuning of at least one environmental characteristic of a first and second acoustic environment.

[0053] A device for obtaining a first distance threshold that at least partially defines an audio transition region enabling adaptive rendering between first and second acoustic environments depending on the listener position in an audio scene can obtain at least one of the following: information indicating the number of acoustic environments to which the first distance threshold is applicable; information indicating the acoustic environments to which the first distance threshold is applicable; information indicating which acoustic environment the first distance threshold is located in; information indicating the distance value relative to the first distance threshold; information indicating the distance value related to the first distance threshold; information indicating the shape or profile associated with the first distance threshold; information indicating a hysteresis offset value that defines a hysteresis region; information indicating an attenuation modulation type that defines a function for correcting the attenuation level when the listener position moves from outside the first distance threshold to the boundary between the first and second acoustic environments; information indicating hops that define the number of hops from the current acoustic environment to the target acoustic environment; and information indicating a parameter attenuation upper limit that defines the maximum attenuation level.

[0054] A device that encodes and outputs information can generate a bitstream comprising encoded information and a first distance threshold.

[0055] The device can be further made to acquire at least one audio signal associated with an audio scene, and the device can be made to generate a bitstream having at least one encoded audio signal, which can then be caused to encode and output the information.

[0056] According to a seventh aspect, an apparatus is provided comprising: means for acquiring information of at least a first acoustic device relating to an audio scene, wherein the audio scene comprises a first acoustic device and a second acoustic device; means for acquiring a first distance threshold that at least partially defines an audio transition region enabling adaptive rendering between the first acoustic device and the second acoustic device depending on the listener position in the audio scene; means for determining the listener position in order to adjust the characteristics of at least one of the first acoustic device and the second acoustic device; and means for adjusting the environmental characteristics of at least one of the first and second acoustic environments according to the listener position, wherein the environmental characteristics are adaptively controlled within the audio scene.

[0057] According to the eighth aspect, there is a means for acquiring information of at least a first acoustic environment related to an audio scene, wherein the audio scene comprises a first acoustic environment and a second environment; means for acquiring a first distance threshold that at least partially defines an audio transition region that enables adaptive rendering between the first acoustic environment and the second acoustic environment depending on the listener position in the audio scene; and means for encoding and outputting the information and the first distance threshold.

[0058] According to the ninth aspect, the device is provided with a computer program comprising instructions [or a computer-readable medium comprising program instructions] for performing at least the following steps: acquiring information about at least a first acoustic environment related to an audio scene, wherein the audio scene comprises a first acoustic environment and a second acoustic environment; acquiring a first distance threshold that at least partially defines an audio transition region that enables adaptive rendering between the first acoustic environment and the second acoustic environment depending on the listener position in the audio scene; determining a listening position in order to adjust the environmental characteristics of at least one of the first acoustic environment and the second acoustic environment; and adjusting the environmental characteristics of at least one of the first acoustic environment and the second acoustic environment depending on the listening position, wherein the environmental characteristics are adaptively controlled within the audio scene.

[0059] According to a tenth aspect, the device is provided with a computer program comprising instructions [or a computer-readable medium comprising program instructions] for performing the following steps: acquiring information on at least a first acoustic environment related to an audio scene, wherein the audio scene comprises a first acoustic environment and a second environment; acquiring a first distance threshold that at least partially defines an audio transition region that enables adaptive rendering between the first acoustic environment and the second acoustic environment depending on the listening position in the audio scene; and encoding and outputting the information and the first distance threshold.

[0060] According to the eleventh aspect, the apparatus is provided with a non-temporary computer-readable medium comprising program instructions for performing: a step of acquiring information on at least a first acoustic environment associated with an audio scene, wherein the audio scene includes the first acoustic environment and the second acoustic environment; a step of acquiring a first distance threshold that at least partially defines an audio transition region that enables adaptive rendering between the first acoustic environment and the second acoustic environment depending on the listening position within the audio scene; a step of determining a listening position to adjust the environmental characteristics of at least one of the first acoustic environment and the second acoustic environment; and a step of adjusting the environmental characteristics of at least one of the first acoustic environment and the second acoustic environment depending on the listening position, wherein the environmental characteristics are adaptively controlled within the audio scene.

[0061] According to a twelfth aspect, the apparatus is provided with a non-temporary computer-readable medium comprising program instructions for causing the apparatus to perform the following steps: acquiring information about at least a first acoustic environment related to an audio scene, wherein the audio scene comprises a first acoustic environment and a second environment; acquiring a first distance threshold that at least partially defines an audio transition region enabling adaptive rendering between the first acoustic environment and the second acoustic environment depending on the listening position within the acquired audio scene; and encoding and outputting the information and the first distance threshold.

[0062] According to a thirteenth aspect, an apparatus is provided comprising: an acquisition circuit configured to acquire information of at least a first acoustic environment related to an audio scene, wherein the audio scene comprises the first acoustic environment and the second acoustic environment; an acquisition circuit configured to acquire a first distance threshold that at least partially defines an audio transition region enabling adaptive rendering between the first acoustic environment and the second acoustic environment depending on the listening position within the audio scene; a determination circuit configured to determine the listening position to adjust the environmental characteristics of at least one of the first acoustic environment and the second acoustic environment; and an adjustment circuit that adjusts the environmental characteristics of at least one of the first acoustic environment and the second acoustic environment depending on the listening position, wherein the environmental characteristics are adaptively controlled within the audio scene.

[0063] According to a fourteenth aspect, an acquisition circuit is provided which is configured to acquire information of at least a first acoustic environment related to an audio scene, wherein the audio scene comprises a first acoustic environment and a second environment; an acquisition circuit which is configured to acquire a first distance threshold which at least partially defines an audio transition region that enables adaptive rendering between the first acoustic environment and the second acoustic environment depending on the listening position in the audio scene; and an encoding and output circuit which is configured to encode and output the information and the first distance threshold.

[0064] According to the 15th aspect, the apparatus comprises the steps of: acquiring information of at least a first acoustic environment related to an audio scene, wherein the audio scene includes the first acoustic environment and the second acoustic environment; A computer-readable medium is provided comprising program instructions for performing the steps of: obtaining a first distance threshold that at least partially defines an audio transition region that enables adaptive rendering between a first acoustic environment and a second acoustic environment depending on the listening position within an audio scene; determining a listening position to adjust the environmental characteristics of at least one of the first acoustic environment and the second acoustic environment; and adjusting the environmental characteristics of at least one of the first acoustic environment and the second acoustic environment depending on the listening position, wherein the environmental characteristics are adaptively controlled within an audio scene.

[0065] According to the 16th aspect, the apparatus is provided with a computer-readable medium comprising program instructions for performing the following steps: acquiring information about at least a first acoustic environment related to an audio scene, wherein the audio scene comprises a first acoustic environment and a second environment; acquiring a first distance threshold that at least partially defines an audio transition region that enables adaptive rendering between the first acoustic environment and the second acoustic environment depending on the listening position within the audio scene; and encoding and outputting the information and the first distance threshold.

[0066] The apparatus includes means for performing the operations described above.

[0067] The apparatus is configured to perform the actions described in the method described above.

[0068] A computer program contains program instructions that cause the computer to perform the methods described above.

[0069] A computer program product stored on a medium can cause the device to perform the methods described herein.

[0070] The electronic device may include the apparatus described herein.

[0071] The chipset may include the apparatus described herein.

[0072] The embodiments of this application aim to address issues related to the latest technology. [Brief explanation of the drawing]

[0073] To better understand this application, please refer here to the attached drawings as an example. [Figure 1] Figure 1 schematically shows several examples of acoustic environments in which several embodiments can be implemented. [Figure 2] Figure 2 schematically shows examples of multiple acoustic environments having acoustic coupling between acoustic environments, in which several embodiments can be implemented. [Figure 3] Figure 3 schematically shows an exemplary system in which several embodiments can be implemented. [Figure 4a] Figures 4a and 4b show flowcharts illustrating the operation of exemplary systems of the apparatus shown in Figure 3, according to several embodiments. [Figure 4b] Figures 4a and 4b show flowcharts illustrating the operation of exemplary systems of the apparatus shown in Figure 3, according to several embodiments. [Figure 5] Figure 5 schematically shows the proximity threshold that enables seamless acoustic environment transition processing of late reverberation to adjacent acoustic environments. [Figure 6] Figure 6 shows an exemplary mapping between the proximity threshold distance and the change in late reverberation decay relative to adjacent acoustic environments when the listener is less than the proximity threshold distance from the boundary between acoustic environments. [Figure 7] Figures 7 and 8 show several exemplary acoustic environments that can be rendered using a constant attenuation rate depending on the proximity distance. [Figure 8]Figures 7 and 8 show several exemplary acoustic environments that can be rendered using a constant attenuation rate depending on the proximity distance. [Figure 9] Figure 9 schematically shows an exemplary device suitable for implementing the device described. [Modes for carrying out the invention]

[0074] As mentioned above, the created or captured audio scenes can be modeled as a series of acoustic environments. For each acoustic environment, a separate reverberation device can be modeled that has characteristics corresponding to the characteristics of the individual acoustic environment (for example, having 15 uncorrelated outputs from output taps of a delay line that models the reverberation).

[0075] The concepts discussed in embodiments of this specification provide apparatuses and methods that enable seamless transitions between two or more acoustic environments. Reverberation modeling for each acoustic environment can be performed differently for each acoustic environment, depending on its acoustic characteristics, such as the physical size of the environment.

[0076] As a result, the renderer can configure reverberation filters individually for each acoustic environment, generating different diffuse delayed reverberation models for different acoustic environments.

[0077] Such a modeling approach creates abrupt changes when the listener moves across the boundary between two acoustic environments. These abrupt changes, along with the distinct differences in diffuse late reverberation between the two acoustic environments, produce an unnatural, distracting, and immersive effect. This is because, in reality, the experience and perception of transitioning from one acoustic environment to another are generally not so dramatic. For example, when moving between a hall and a corridor, the audio doesn't suddenly change in a way that's disruptive.

[0078] Accordingly, embodiments described herein provide apparatus and methods for enabling smooth and seamless transitions that make the listening experience more trustworthy.

[0079] While diffuse late reverberation is inherently ignorant of the listener's position within the acoustic environment in some embodiments, the resulting reverberation rendering is modified to be more believable in specific situations (such as near the boundaries of the acoustic environment).

[0080] In some embodiments, the apparatus and method thus relate to obtaining a seamless transition between acoustic environments (AEs), and a mechanism is provided for modifying delayed reverbation in a first AE depending on the listener position adjacent to a second acoustically coupled AE, which is performed to achieve a resulting delayed reverb rendering with a perceptually seamless transition between two acoustic environments. The perceptually seamless transition, in some embodiments, involves the steps of: obtaining acoustic environment extent information from audio scene metadata in a bitstream; obtaining proximity threshold distances to other AEs in audio scene metadata in a bitstream; determining the listener's current acoustic environment depending on the listener's position; This can be achieved by setting the current acoustic environment so that the reverberation decay is zero. In this way, the reverberation is rendered at the maximum gain level.

[0081] Determine whether the proximity of the listener to other AEs is within the acquired proximity threshold distance.

[0082] For all AEs within each proximity threshold distance, the delayed reverberation attenuation is adjusted (linearly) to the proximity threshold distance for each AE. For example, if the proximity threshold distance is 1m and the listener is 90cm from the second AE, the reverberation attenuation value for the second AE will be set to 90%.

[0083] Late reverberation can be rendered for all AEs according to their respective determined decay values.

[0084] Then, all the reverberations are added together to obtain the resulting late reverberation.

[0085] In some embodiments, the proximity threshold is generated in proportion to the difference between one or more of the following: acoustic environment parameters such as RT60, the size of the acoustic environment, and the size of the acoustic coupling (e.g., width and height, width, height, and length).

[0086] In some embodiments, the variation in attenuation values ​​when a listener is within a proximity threshold distance to an adjacent acoustically coupled AE can vary according to a type defined within the audio scene rendering bitstream. The type of variation can be linear, exponential, etc.

[0087] In some further embodiments, the signaling allows the maximum attenuation to be less than 100%. This results in a particular acoustic environment being completely absent from the audio scene over time. Such parameters can be a simple way to diffuse the sound of delayed reverberation of a particular AE throughout the scene without the need to specifically incorporate reverberation routing through portals. This can be of significant benefit in complex scenes with coupled acoustic environments (e.g., including multiple AEs with acoustic coupling between them).

[0088] Acoustic transition information for implementing seamless transitions with crossfades can be signaled via a bitstream to modify late reverberation transition behavior across acoustic acoustics (AEs). In some embodiments, such signals may comprise individual parameters in the bitstream or signal indices in a predefined table.

[0089] In some embodiments, the proximity threshold may be an area or a volume.

[0090] With respect to Figures 1 and 2, exemplary audio scenes are shown. The audio scene comprises a door 104 area, a first acoustic environment AE1100, and a second acoustic environment AE2102. The second acoustic environment AE2102 is shown with two audio sources, namely the first audio source S1105 and the second audio source S2103. Furthermore, it is possible to move freely twice within the audio scene to a first position P1115 in the second acoustic environment AE2102, a second position P2113 in the first acoustic environment AE1100, And a listener is shown at a third position P3111 when the listener is outdoors 104.

[0091] The difference between the audio scenes shown in Figures 1 and 2 is the presence of a known acoustic coupling AC1200 between the first acoustic environment AE1100 and the second acoustic environment AE2102, where the second acoustic environment AE2102 defines the "free" audio exchange between the two AEs.

[0092] Each acoustic environment is modeled and configured to reproduce late reverberation according to specified acoustic characteristics. These characteristics include parameters such as RT60 and diffusion-to-direction ratio (DDR). The values ​​of the DDR parameters can be assigned either based on a specific calculation approach or as specified by the content creator's intent.

[0093] For example, known methods for determining or calculating DDR are described in Section 3.3 of MPEG-I I Immersive Audio CfP Supplemental Information, Recommendations and Clarifications, Version 1, N00083. DDR is also defined in Section 3.9 EIF N00054.

[0094] As described above, the audio scene is shown to include two active audio sources S1105 and S2103. In addition, an acoustic coupling AC1 (or opening) connecting the acoustic environments AE1100 and AE2102 allows audio energy to be transferred from AE2102 to AE1100.

[0095] The presence and details of acoustic coupling information can be obtained or determined based on any suitable method. For example, this information can be determined by geometric analysis, which involves photographing a setting with densely packed rays in all directions to identify an aperture. However, methods for determining the presence and degree of acoustic coupling are not described in further detail below. In this invention, acoustic coupling, which listeners are expected to transition to, is more important because it requires special rendering to provide a seamless transition between AEs.

[0096] As mentioned above, it is important to enable a seamless transition in late reverberation rendering when the listener moves between two acoustic environments.

[0097] Figures 3 and 4 show schematic diagrams of apparatus suitable for implementing several embodiments, as well as flowcharts of embodiments. Therefore, the implementation of the reverberation embodiment after seamless transition aims to avoid abrupt changes that would impair the validity and consequent immersion in a 6DoF audio scene.

[0098] As shown in Figure 3, there is an exemplary creation device 301 configured to acquire content in the form of virtual scene definition parameters and audio signals, and to provide a suitable bitstream / data file containing the audio signals and virtual scene definition parameters.

[0099] In some embodiments, as shown in Figure 3, the creation device 301 includes an encoder input format (EIF) data generator 311. The encoder input format (EIF) data generator 311 generates EIF (Encoder Input Format) data, which is a content creator scene description. The scene description information includes virtual scene geometry information, such as the positions of audio elements. Furthermore, the scene description information may include other relevant metadata, such as directivity and size, as well as other acoustically relevant elements. For example, the relevant metadata may include the positions of virtual walls, their acoustic properties, and other acoustically relevant objects such as occluders. Examples of acoustic properties are acoustic material properties such as (frequency-dependent) absorption or reflection coefficients, the amount of scattering energy, or transmission properties. In some embodiments, the virtual acoustic environment can be described according to its (frequency-dependent) reverberation time or diffusion-to-direct sound ratio. In some embodiments, the EIF data generator 311 may be more generally known as a virtual scene information generator. The virtual scene description may, in some embodiments, include several acoustic environment descriptions. In some embodiments, virtual scene reverberation parameters are derived based on reverberation characterization information such as a pre-delay specifying the time required for the audio signal to decay 60 dB below its initial level, a -60 dB reverberation time (RT60), or a diffuse-to-direct ratio (DDR) specifying the level of diffuse reverberation relative to the level of total radiated sound in each of the acoustic environment descriptions specified in the EIF. RT60 and DDR are frequency-dependent properties. EIF parameters 312 may be provided to a suitable bitstream encoder 317 in some embodiments.

[0100] As will be described in more detail in this application, one of the parameters used in late reverberation and seamless late reverberation transition bitstream generation is the proximity threshold distance. The proximity threshold distance is the distance at which a listener can hear reverberation from an adjacent AR. The proximity threshold distance may be derived programmatically or based on the creative intent of the content creator.

[0101] In some embodiments, the proximity threshold distance can be derived from the following equation.

number

[0102] In the above formula, DTH corresponds to the listener proximity threshold in the current AE (also called the source AE), and the neighboring AE to which the listener is transitioning (e.g., by walking or teporting) is called the destination AE. RT60D and RT60S represent the RT60 of the destination and source AEs. The values ​​of DDRD and DDRS represent the DDR values ​​of the destination and source AEs. VD and VS represent the capacities of the destination and source AEs. K1, K2, and K3 can be specified by the encoder depending on the scene data. The above is merely one example of deriving the proximity distance. This distance may be derived by the content creator through manual tuning and specified in the encoded bitstream. In other embodiments, the proximity distance may be renderer implementation selectivity.

[0103] In some embodiments, the creator device 301 includes an audio content generator 313. The audio content generator 313 is configured to generate audio content corresponding to an audio scene. In some embodiments, the audio content generator 313 is configured to generate, or otherwise acquire, audio signals associated with a virtual scene. For example, in some embodiments, these audio signals may be acquired or captured using a suitable microphone or array of microphones, based on processed acquired audio signals, or synthesized. In some embodiments, the audio content generator 313 is further configured to generate or acquire audio parameters related to the audio signals, such as position or signal directivity within the virtual scene. The audio signals and / or parameters 312 may, in some embodiments, be provided to a suitable audio encoder 315 and bitstream encoder 317.

[0104] The creator device 301 may further include a suitable audio encoder 315. In some embodiments, the audio encoder 315 is configured to use an audio signal and generate encoder audio which is passed to the bitstream encoder 317.

[0105] In some embodiments, the creator device 301 includes a bitstream encoder 317. The bitstream encoder 317 is configured to receive EIF parameters 312 and encoded audio signals / parameters and to generate an appropriate encoded bitstream based on this information. This may be, for example, an MPEG-I 6DoF audio bitstream. In some embodiments, the bitstream encoder 317 can be a dedicated encoding device. The output of the bitstream encoder 317 can be passed to distribution or storage device 303. In one embodiment, the audio signals in the MPEG-I 6DoF audio bitstream can be encoded in the MPEG-H 3D format described in "ISO / IEC 23008-3:2018 High efficiency coding and media delivery in heterogenous environments - Part 3: 3D audio". This specification describes appropriate encoding methods for audio objects, channels, and higher-order ambisonics. The low complexity (LC) profiles described herein may be particularly useful for encoding audio signals.

[0106] In some embodiments, the following is an example of a bitstream syntax that can be incorporated into the delayed reverberation rendering metadata.

number

[0107] Regarding AcousticEnvironmentTransitionStruct(), the num_AcousticEnvironments parameter is configured to provide the number of AEs that a listener can pass through via acoustic coupling. The included AEs are not limited to adjacent neighbor objects only, but may also be limited to AEs that are acoustically coupled to adjacent neighbor objects.

[0108] The `current_acousticEnvironment_ID` parameter provides the environment ID for the current AE. This may not exist if `AcousticEnvironmentTransitionStruct()` is delivered within `LateReverbStruct()`.

[0109] The destination_acousticEnvironment_ID can be configured to provide the environment ID of the destination AE that the listener can traverse. The transition can occur either by simple walking or teleportation.

[0110] Furthermore, the proximity_threshold_distance parameter is configured to identify the distance (in millimeters) required to initiate a seamless transition of late reverberation from a neighbor when the listener is near the boundary between AEs.

[0111] The parameter num_hops can define the number of hops from the current AE to the destination AE. In the immediate vicinity, the hop size is 1. Furthermore, when the listener is near the boundary between AEs, there is a value distance (in millimeters) for initiating a seamless transition of late reverberation from the neighbor.

[0112] Furthermore, the parameter hysteresis_offset is the distance (in millimeters) added to proximity_threshold_distance after the listener has moved a shorter distance than proximity_threshold_distance to other AEs in the audio scene. This value is added to prevent unnatural behavior when the user moves only a small amount forward or backward at the proximity_threshold_distance boundary. In other embodiments, hysteresis_offset can be specified as a percentage point in the range of 1 to 100.

[0113] In some embodiments, `attenuation_modulation_type` identifies a method for correcting the attenuation level from 100 to 0 as the listener moves from outside the proximity threshold distance to the boundary. The value can indicate, for example, the type of curve used, such as linear or exponential.

[0114] The parameter `attenuation_upperbound_index` represents the maximum attenuation level. A refractive index value of 0 indicates a maximum attenuation level of 100%. For index values ​​greater than 1, the maximum attenuation is limited to 95%, 90%, etc.

[0115] The seamless transition data structure described above can be embedded in LateReverbStruct() in some embodiments as described below.

number

number

[0116] In some embodiments of the LateReverbStruct() structure, the parameter numberOfSpatialPositions defines the number of delay line positions in the delayed reverberation payload. This value is defined using indices corresponding to a specific number of delay lines. The bit string value "0b00" signals the renderer to 15 spatial values ​​for the delay line. The other three values, "0b01", "0b10", and "0b11", are already reserved.

[0117] In some embodiments, the orientation parameter defines the orientation of the delay line relative to the listener. The range is -180 to 180 degrees.

[0118] The parameter elevation angle can identify the elevation angle that defines the elevation angle of the delay line relative to the listener. The range is -90 to 90 degrees.

[0119] The parameter `numberOfAcousticEnvironments` defines the number of acoustic environments in the audio scene. In some embodiments, `LateReverbStruct()` carries information about one or more acoustic environments present in the audio scene at that point in time.

[0120] Furthermore, the parameter environmentId defines a unique identifier for the acoustic environment.

[0121] The delayLineLength value defines the sample-level length of the graphic equalizer (GEQ) filter used to configure the attenuation filter. The lengths of different delay lines corresponding to the same acoustic environment are relatively prime.

[0122] The structure filterParamsStruct() can, in some embodiments, describe a graphic equalizer cascade filter for configuring a decay filter for delay lines. The same structure is then used to configure a filter for diffuse-direct reverberation ratio.

[0123] The filterParamsStruct()SOSLength parameter is the length of each of the quadratic section filter coefficients. The values ​​of b1, b2, a1, and a2 are the filter coefficients b1, b2, a1, and a2. Furthermore, the globalGain parameter allows you to specify the GEQ gain coefficient in decibels.

[0124] The levelDB value specifies the sound level in decibels for each delay line.

[0125] In other implementations, the parameters fadeInDistance and the seamless transition enable flag may be present in the renderer configuration. In other embodiments, these may be included in the bitstream as follows:

number

[0126] The method for selecting or generating AE parameters can be illustrated with reference to the flowchart in Figure 4a.

[0127] Therefore, for example, step 451 yields a list of all AEs as shown in Figure 4a.

[0128] Next, step 453 determines / acquires or receives the acoustic coupling between the AEs, as shown in Figure 4a.

[0129] Next, step 455 obtains the parameters for each set of acceptable AE transitions in the audio scene for one of the AEs, as shown in Figure 4a. (Furthermore, in some embodiments, one of the AEs is set as the "current AE".)

[0130] Step 457 determines the proximity threshold for other AEs relative to the current AE, as shown in Figure 4a.

[0131] Next, in step 459, a proximity threshold distance is inserted into the late reverberation payload parameter of the current AE to enable a seamless transition between a specific current AE and other AEs, as shown in Figure 4a.

[0132] Next, in step 461, this method is repeated for all acoustically coupled AEs with respect to the current AE, as shown in Figure 4a.

[0133] In the iterative loop, the proximity threshold distance is determined by step 463 for each acoustically coupled AE with respect to all other acoustically coupled AEs, as shown in Figure 4a.

[0134] Furthermore, the system of the apparatus shown in Figure 3 includes an (optional) storage / distribution device 303. The storage / distribution device 303 is configured to retrieve encoded bitstreams 316 (including parameters and audio signals) from the creation device 301 and store and / or distribute them to the appropriate playback device 305. In some embodiments, the functions of the storage / distribution device 303 are integrated into the creator device 301.

[0135] In some embodiments, the bitstream is delivered over a network having any desired delivery format. Exemplary delivery formats that may be employed in some embodiments are any suitable approach such as DASH (Dynamic Adaptive Streaming over HTTP), CMAF (Common Media Application Format), or HLS (HTP live streaming).

[0136] In some embodiments, which will not be described in further detail in this example, the audio signal is sent to the encoding parameters in a separate data stream.

[0137] In some embodiments, the storage / distribution device 303 includes a bitstream storage device 321 configured to acquire and store encoded audio and parameters.

[0138] Furthermore, in some embodiments, the storage / distribution device 303 includes a content selector / supplier 323 configured to distribute an encoded bitstream (encoded audio and parameters). In some embodiments, the content selector / supplier is configured to supply metadata in the form of encoded parameters and audio data 324 based on listener location information, as shown in Figure 3.

[0139] The system of the apparatus shown in Figure 3 further comprises a playback device 305. The playback device 305 is configured to acquire encoded parameters and encoded audio signals 324 (based on listener position) from the storage / distribution device 303. In addition, in some embodiments, the playback device 305 is configured to acquire sensor data 330 (related to the physical listening space) and to generate one or more appropriate rendered audio signals 334 to be provided to the user of a head-mounted device HMD 307 (which may comprise sensors and headphones in some embodiments), for example, as shown in Figure 3. The sensors can determine, for example, the physical space in which the listener is located, or the listener's position and / or orientation.

[0140] In some embodiments, the playback device 305 includes a (MPEG-I 6DoF) player 321 configured to receive a bitstream 324 (including parameters and audio data). In some embodiments, for AR rendering, the device is also expected to include an AR sensing module to obtain the physical characteristics of the listening space.

[0141] A 6DoF bitstream (including parameters and audio signals) alone is sufficient to perform rendering in VR scenarios. That is, in VR scenarios, The necessary acoustic information is transported via a bitstream, which is sufficient to render the audio scene at different virtual locations within the scene according to virtual acoustic characteristics such as material and reverberation parameters.

[0142] In AR scenarios, the renderer can acquire listener spatial information during rendering using AR sensing provided to the renderer, for example in LSDF format. This provides information about the listener's physical spatial reflective elements (walls, curtains, windows, openings between rooms, etc.).

[0143] Therefore, for example, in some embodiments, a user or listener operates (or wears) a suitable head-mounted device (HMD) 207. The HMD may be equipped with sensors configured to generate suitable sensor data 330 that can be passed to a playback device 305.

[0144] The playback device 305 (and the MPEG-I 6DoF player 321) further includes, in some embodiments, an AR sensor analyzer 331. The AR sensor analyzer 331 is configured to generate physical spatial information (from HMD sensing data or otherwise). This could be, for example, an LSDF parameter format and the associated LSDF parameters passed to a suitable renderer 333.

[0145] In some embodiments, the playback device 305 (and MPEG-I 6DoF player 321) further comprises a content selector 342. In some embodiments, the content selector 342 is configured to acquire the listener position and / or orientation and pass this information to the content selector / supplier 323 so that the content selector / supplier 323 can provide audio and parameters based on the listener position.

[0146] The playback device 305 (and MPEG-I 6DoF player 321) further includes, in some embodiments, a renderer 333 configured to receive virtual space parameters, audio signals (and in some embodiments, physical listening space parameters), and generate a suitable spatial audio signal to be output to the HMD 307 as shown in Figure 3, for example, as a binaural audio signal output by headphones.

[0147] In some embodiments, the renderer 333 includes a geometry checker 334. In some embodiments, the geometry checker 334 is configured to determine or obtain a listener position and, based on the listener position, to check whether the listener position places the listener in one of the acoustic environments, and if so, which acoustic environment it is in. This acoustic environment can be determined as a current AE.

[0148] The geometry checker 334 can also be configured to perform a geometric check to determine whether the listener position is within the vicinity of a neighboring AE, based on the listener position, and the proximity threshold is obtained from the delayed reverberation rendering metadata received during 6DoF rendering bystream.

[0149] In some embodiments, the renderer 333 includes a current AE reverberator 336. The current AE reverberator 336 is a reverberator associated with the current AE and can be implemented according to any suitable method. For example, in some embodiments, the reverberator 336 is implemented as a delay line having a variable gain feedback loop or a variable gain feedforward loop.

[0150] In some embodiments, the renderer 333 includes a destination AE reverberator 338. The destination AE reverberator 338 is a reverberator associated with the destination AE and can be implemented according to any suitable scheme. For example, in some embodiments, the reverberator 338 is implemented as a delay line having a variable gain feedback loop or a variable gain feedforward loop. In some embodiments, there may be other AEs that contribute to the current and destination reverberation. It is implicitly assumed that such AEs are included in the rendering for the current and destination AE delayed reverberation rendering.

[0151] In some embodiments, the renderer 333 includes a delayed reverberation mixer 332. The delayed reverberation mixer is configured to take current and the output from the delayed reverberator of the destination delayed renderer, combine them, and produce a combined rendering output.

[0152] Therefore, generally, seamless transitions are implemented in the following way: If it is determined that a listener is within the boundary of a particular AE extent, the listener is set to be within that AE. For example, in a scene containing acoustic environments of AE1 and AE2, if the listener is within the boundary of AE1, it is set to be within AE1. Subsequently, if the renderer notices that the listener is within AE1 but is further away from the proximity threshold distance to the decay of AE2 and AE1, the decay of AE2 is set to 0 and then to 100 for seamless transition handling. This is because the listener is quite far from the transition zone. Subsequently, a seamless transition mechanism is effective in rendering the decay of AE1 to 0 and the decay of AE2 according to the proximity distance (below the proximity threshold) so that the listener's position remains within AE1, even though the listener is less than the proximity threshold distance to AE2. In this way, the reverberation, which is the sum of the late reverberation of AE1 with decay of 0 and the late reverberation of AE2 with decay of less than 100 but greater than or equal to 0, is seamlessly rendered in transition. This continues until the listener moves to the boundary between AE1 and AE2. At this point, the attenuation becomes 0 for both AE1 and AE2.

[0153] The operations performed by the renderer for several embodiments (which will be described in more detail above) are shown in the flowchart in Figure 4b.

[0154] Therefore, by step 401, information about all AEs in the audio scene is obtained, as shown in Figure 4b. In some embodiments, this information is obtained in the form of 6DoF metadata in a bitstream corresponding to the 6DoF audio scene currently being consumed. Thus, the information comprises information for one or more AEs in the audio scene. In some embodiments, the audio scene includes two or more AEs. Furthermore, in some embodiments, the audio scene further includes acoustic coupling between AEs, which allows the listener to transition across acoustic environments. In addition, late reverberation rendering parameters are obtained for one or more AEs in the audio scene.

[0155] The decay level of all AEs in the audio scene is set to full or 100%. In other words, by step 403, all late reverberation renderings for all AEs are set not to produce sound, as shown in Figure 4b.

[0156] Next, in step 405, a geometric check is performed based on the listener's current position within the audio scene to identify the AE where the listener is located, as shown in Figure 4b.

[0157] For AEs that are determined to have a listener, step 407 sets the attenuation level to 0, as shown in Figure 4b.

[0158] Therefore, the current AE delayed reverberation is set to full volume by step 409, as shown in Figure 4b.

[0159] In the next step, the renderer can be configured to obtain or look up a listener proximity threshold for each of the other AEs (e.g., neighboring AEs) that it has determined will allow the listener to transition through the AE boundary during scene consumption. Obtaining the proximity threshold is shown in Figure 4b by step 411.

[0160] An example of proximity thresholding is shown in Figure 5. In this example, the first and second acoustic environments are shown as AE1500 and AE2510. There is an acoustic coupling 520 between the acoustic environments. Furthermore, a proximity threshold DTH-AE1-AE2501 is shown within AE1 and associated with AE2. The proximity threshold DTH-AE1-AE2501 is located at position 505, away from the acoustic coupling 520 within AR1. Furthermore, a proximity threshold DTH-AE2-AE1503 is shown within AE2 and associated with AE1. The proximity threshold DTH-AE2-AE1503 is located at position 507, further away from the sound wave coupling 520 within AE2510. The proximity threshold defines the seamless transition processing threshold marker 511.

[0161] Step 413, as shown in Figure 4, provides a proximity threshold for the neighbor AE to the listener, obtained from the received delayed reverberation rendering metadata within the 6DoF rendering bistream.

[0162] If, based on determining the proximity of the neighboring AEs in the geometrical check, at least one neighboring AE is determined, a further geometrical check is performed, as shown in step 413 in Figure 4. Attenuation in some embodiments is modulated or modified according to the proximity of the listener to the neighboring AE.

[0163] In some embodiments, attenuation modulation or modification is performed linearly such that the attenuation is set to 100 when the listener distance is greater than the proximity threshold to other AEs. On the other hand, when the listener is at a distance of 0 from other AEs, the attenuation is set to 0.

[0164] This initial linear setting is shown in Figure 4b by step 415.

[0165] The variation between 0 and 100 can be controlled based on any appropriate mapping or function. Furthermore, in some embodiments, this mapping or function is determined based on parameters delivered from the bitstream that specify whether the variation is linear, exponential, etc. The modification or mapping function applied to the gain value is shown in Figure 4b by step 417.

[0166] Referring to Figure 6, exemplary linear mappings 600 and nonlinear mappings 610 are shown. Thus, as shown with respect to the example of linear mapping 607, the attenuation 601 from 0% to 100% varies from a distance of 0 to a proximity threshold distance 605 from another AE 603. An exemplary linear (or exponential) mapping 617 is also shown, in which the attenuation 611 from 0% to 100% varies from a distance of 0 to a proximity threshold distance 615 from another AE 613.

[0167] Next, once the gain is set, step 419 renders the AE according to the respective attenuation values, as shown in Figure 4b.

[0168] Once the reverberation rendering portion is generated, step 421 sums up all AE portions as shown in Figure 4b.

[0169] Next, step 423 outputs a rendering as shown in Figure 4b.

[0170] Note that in the case of more than two acoustic amplifiers (AEs) within an audio scene, acoustic coupling may exist between multiple AEs. For example, as shown in Figure 7, there is a first acoustic coupling AC1702 between AE1700 and AE2710. The first, second, and third acoustic environments AE1700, AE2710, and AE3720 are shown, with a second acoustic coupling AC2712 between AE2710 and AE3720. Furthermore, the first proximity threshold DTH-AE1-AE2701 within AE1700 and the second proximity threshold DTH-AE1-AE3703 located within AE1700 are shown and associated with AE3.

[0171] In such cases, the proximity threshold is applied to listener L 705 in AE1700, corresponding to AE2710 and AE3720. The difference from the previous example is that the attenuation of AE3 does not become zero when transitioning from AE1 to AE2. This is because the attenuation of AE3 in the late reverberation becomes zero when the listener is at the boundary with AE2. The advantage of this method is that it does not require supplying AE3 to AE2 and then mixing AE2 with AE1.

[0172] Therefore, such embodiments reduce the overall processing delay induced by supplying cascaded coupled AEs before performing delayed reverberation rendering output.

[0173] Furthermore, this approach can be applied to other geometries, as shown in Figure 8, for audio scenes containing four acoustic environments. In this example, the first, second, third, and fourth acoustic environments AE1800, AE2810, AE3820, and AE4830 are shown, where the first acoustic coupling AC1802 is between AE1800 and AE2810, the second acoustic coupling AC2812 is between AE2810 and AE3820, and the third acoustic coupling AC3822 is between AE1800 and AE4830. Furthermore, within AE1800, a first proximity threshold DTH-AE1-AE2801 related to AE2, a second proximity threshold DTH-AE1-AE3811 located within AE1800 and related to AE3, and a third proximity threshold DTH-AE1-AE4821 located within AE1800 and related to AE3 are shown.

[0174] In such cases, the geometry check is configured to identify the listener's movement toward the relevant transition. For example, the transition AE1->AE2 compared to AE1->AE4 results in various attenuation filter choices.

[0175] Therefore, in some embodiments where geometry checks are configured to determine the transition from AE1 to AE2, AE3 may also be rendered during approach from listener to AE2 to enable a seamless transition from AE1 to AE2, in addition to rendering AE2. However, the proximity threshold for AE3 must be extended to the listener's position within AE1.

[0176] Similarly, in some embodiments where the listener transitions from AE1->AE4, only AE4 is rendered, allowing a seamless transition to AE4 within the proximity threshold.

[0177] In some embodiments, the proximity threshold can be implemented as a proximity region (which may be specified by an area or volume). The region or volume may be a primitive. In some implementations, the proximity volume may also be specified by a mesh.

[0178] Furthermore, as discussed herein, there are devices and possible mechanisms that provide practical rendering for immersive audio within AR applications.

[0179] The embodiments described herein combine listening space characteristics and virtual scene rendering parameters to obtain fused rendering that provides appropriate audio performance regardless of scene characteristics.

[0180] The fusion (or combination) described in some embodiments is implemented such that the auralization does not recognize whether the rendering is for AR or VR. In other words, the embodiments described herein can be implemented within a system suitable for performing AR, VR (and mixed reality (MR)). Such a mechanism allows AR rendering to be deployed using many different auralization implementations.

[0181] In some embodiments, the apparatus and possible mechanisms described herein have 6 degrees of freedom. Binaural rendering of audio can be performed in a system that has (i.e., the listener or listening position can move within the scene and the listener position is tracked).

[0182] In such embodiments, an apparatus and method are proposed that use information from an audio scene specified by a bitstream consisting of a virtual scene description and a description of the listener's physical space obtained during rendering to enable the audibility of the virtual and physical spaces, thereby obtaining a unified scene representation that brings high quality immersion within the physical space.

[0183] With respect to Figure 9, an exemplary electronic device (e.g., computer 1 511, computer 2 521, or computer 3 531) may represent any of the devices shown above. The device may be any suitable electronic device or apparatus. For example, in some embodiments, device 1400 is a mobile device, user equipment, tablet computer, computer, audio playback device, etc.

[0184] In some embodiments, the device 1400 comprises at least one processor or central processing unit 1407. The processor 1407 can be configured to execute various program codes, such as those described herein.

[0185] In some embodiments, the device 1400 includes a storage device 1411. In some embodiments, at least one processor 1407 is coupled to the storage device 1411. The storage device 1411 can be any suitable storage means. In some embodiments, the storage device 1411 includes a program code section for storing program code that can be executed on the processor 1407. Furthermore, in some embodiments, the storage device 1411 may further include a stored data section for storing data, for example, data that has been processed or is to be processed according to the embodiments described herein. The executed program code stored in the program code section and the data stored in the stored data section can be retrieved by the processor 1407 via a memory-processor coupling as needed.

[0186] In some embodiments, the device 1400 includes a user interface 1405. The user interface 1405 may be coupled to the processor 1407 in some embodiments. In some embodiments, the processor 1407 can control the operation of the user interface 1405 and receive input from the user interface 1405. In some embodiments, the user interface 1405 can allow a user to input commands to the device 1400, for example, via a keypad. In some embodiments, the user interface 1405 can allow a user to retrieve information from the device 1400. For example, the user interface 1405 may include a display configured to show information from the device 1400 to the user. In some embodiments, the user interface 1405 may include a touchscreen or touch interface capable of both allowing information to be input to the device 1400 and further displaying the information to the user of the device 1400. In some embodiments, the user interface 1405 may be a user interface for communicating with a position determiner, as described herein.

[0187] In some embodiments, the device 1400 includes an input / output port 1409. In some embodiments, the input / output port 1409 includes a transceiver. The transceiver in such embodiments may be coupled to a processor 1407 and configured to enable communication with other devices or electronic devices, for example, via a wireless communication network. The transceiver or any preferred transceiver or transmitter and / or receiver means may, in some embodiments, be configured to communicate with other electronic devices or equipment via a wired connection or wired coupling.

[0188] The transceiver can communicate with further devices using any suitable known communication protocol. For example, in some embodiments, the transceiver may use a suitable Universal Mobile Telecommunications System (UMTS) protocol, a Wireless Local Area Network (WLAN) protocol such as IEEE 802.X, a suitable short-range radio frequency communication protocol such as Bluetooth®, or an Infrared Data Pathway (IRDA).

[0189] The transceiver input / output port 1409 can be configured to receive signals. In some embodiments, the parameters are determined as described herein by using a processor 1407 that executes appropriate code.

[0190] Furthermore, while the embodiments described above are exemplary, it should be noted that there are several variations and modifications that can be made to the disclosed solutions without departing from the technical scope of the present invention.

[0191] In general, various embodiments can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some embodiments of this disclosure may be implemented in hardware, and others may be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device, but this disclosure is not limited to these. Various embodiments of this disclosure may be illustrated and intended using block diagrams, flowcharts, or any other graphic representation, but it will be fully understood that these blocks, devices, systems, techniques, or methods intended herein may, in non-limiting examples, be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers, or other computing devices, or any combination thereof.

[0192] As used in this application, the term “circuit” may mean one, more or all of the following: (a) Hardware-only circuit implementations (such as analog and / or digital circuit-only implementations) and (b) A combination of hardware circuits and software, such as (if applicable), (i) combinations of analog and / or digital hardware circuits and software / firmware, (ii) Any part of a hardware processor having software, software and storage devices (including a digital signal processor) Collaborate to enable devices such as mobile phones or servers to perform various functions. (c) Hardware circuits and processors that require software (e.g., firmware) to operate, such as microprocessors or parts of microprocessors, may not have software when it is not required for operation.

[0193] This definition of circuit applies to all use of the term in this application, including in any of the claims. Further examples include, as used in this application, the term circuit also encompasses not only a hardware circuit or processor (or more processors), but also a portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware implementation.

[0194] The term "circuit" also includes, for example, baseband integrated circuits or processor integrated circuits for mobile devices or similar integrated circuits in servers, cellular network devices, or other computing or network devices, where applicable to a particular claim element.

[0195] Embodiments of the present disclosure may be implemented by computer software executable by a data processor in a mobile device, such as within a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs, also called program products, which include software routines, applets, and / or macros, may be stored on any device-readable data storage medium and comprise program instructions for performing a particular task. A computer program product may comprise one or more computer executable components configured to perform embodiments when the program is executed. One or more computer executable components may be at least one piece of software code or a portion thereof.

[0196] Furthermore, it should be noted that in this regard, any block in the logic flow shown in the diagram may represent a program step, or an interconnected set of logic circuits, blocks, and functions, or a combination of program steps and logic circuits, blocks, and functions. Software can be stored on physical media such as memory chips, or memory blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variant CDs. Physical media are non-temporary media.

[0197] The memory can be of any type suitable for the local technical environment and can be implemented using any suitable data storage technology, such as semiconductor-based memory, magnetic memory and systems, optical memory and systems, fixed memory and removable memory. The data processor may be of any type suitable for the local technical environment and may comprise, in non-limiting examples, one or more of the following: general-purpose computers, dedicated computers, microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), FPGAs, gate-level circuits, and processors based on multicore processor architectures.

[0198] Embodiments of this disclosure can be implemented in various components, such as integrated circuit modules. Integrated circuit design is performed through large-scale, highly automated processes. Complex and powerful software tools are available to translate logic-level designs into semiconductor circuit designs that are ready to be etched and formed on a semiconductor substrate.

[0199] The scope of protection required for the various embodiments of this disclosure is indicated by the independent claims. Embodiments and features described herein but not within the technical scope of the independent claims should be construed as useful examples for understanding the various embodiments of this disclosure.

[0200] The foregoing description has provided a complete and informative description of exemplary embodiments of the present disclosure, as non-limiting examples. However, various modifications and adaptations will become apparent to those skilled in the art upon careful reading of the accompanying drawings and claims, taking the foregoing description into consideration. Nevertheless, all such and similar modifications of the teachings of the present disclosure remain within the scope of the invention as defined in the accompanying claims. In fact, there are further embodiments, including one or more embodiments combined with any of the other embodiments discussed above.

Claims

1. An apparatus for enabling audio transitions between at least two acoustic environments, the apparatus comprising means configured to acquire information of at least a first acoustic environment relating to an audio scene, wherein the audio scene comprises a first acoustic environment and a second acoustic environment, acquire a first distance threshold that at least partially defines an audio transition region enabling adaptive rendering between the first and second acoustic environments depending on the listening position within the audio scene, determine a listening position to adjust the environmental characteristics of at least one of the first and second acoustic environments, and adjust the environmental characteristics of at least one of the first and second acoustic environments depending on the listening position so that they are adaptively controlled within the audio scene.

2. The apparatus according to claim 1, wherein the means is configured to acquire information of at least a first acoustic environment relating to the audio scene, wherein the audio scene includes the first acoustic environment, and the second acoustic environment is configured to receive a bitstream containing information of at least the first acoustic environment relating to the audio scene.

3. The method of the apparatus according to claim 2, wherein the bitstream further comprises the first distance threshold, and the means configured to acquire the first distance threshold is configured to acquire the first distance threshold from the bitstream.

4. The apparatus according to any one of claims 1 to 3, wherein the first acoustic environment and the second acoustic environment are coupled by a first acoustic coupling located at the boundary between the first acoustic environment and the second acoustic environment.

5. The first distance threshold that defines the audio transition region at least partially is The apparatus according to any one of claims 1 to 4, which is located within the first acoustic environment and associated with the second acoustic environment.

6. The apparatus according to any one of claims 1 to 5, wherein the means configured to adjust the environmental characteristics of at least one of the first and second acoustic environments in accordance with the listening position is configured to determine that the listener position is within an audio transition region and to adjust at least one of a first reverberation gain parameter of the first acoustic environment or a second reverberation gain parameter of the second acoustic environment based on the listening position relative to the first distance threshold and the boundary between the first acoustic environment and the second acoustic environment.

7. The apparatus according to claim 6, wherein the means is further configured to acquire a first function related to the first distance threshold, and is configured to adjust the environmental characteristics of at least one of the first and second acoustic environments in accordance with the listening position, and is further configured to adjust at least one of a first reverberation gain parameter of the first acoustic environment or a second reverberation gain parameter of the second acoustic environment based on the first function applied to the listening position with respect to the first distance threshold and the boundary between the first acoustic environment and the second acoustic environment.

8. The apparatus according to any one of claims 1 to 7, wherein the means is further configured to render a spatial audio signal, the spatial audio signal having at least partially generated reverberation based on the environmental characteristics of at least one of the first and second acoustic environments.

9. The apparatus according to claim 8, wherein the means configured to render the spatial audio signal is configured to generate the spatial audio signal having a first reverberation portion based on a first reverberation gain parameter of the first acoustic environment, and / or to generate a second reverberation portion based on a second reverberation gain parameter of the second acoustic environment.

10. The apparatus according to claim 9, wherein the means configured to generate the spatial audio signal comprising a first reverberation portion based on a first reverberation gain parameter of the first acoustic environment and / or a second reverberation portion based on a second reverberation gain parameter, is configured to set the current acoustic environment as the first acoustic environment when the listener position is within the first acoustic environment, set reverberation attenuation for all other acoustic environments except the first acoustic environment which does not contribute, set reverberation attenuation to provide a defined contribution for the second acoustic environment which is within a region of the first acoustic environment defined by the threshold, determine the reverberation contributions to the current acoustic environment and other acoustic environments, and combine the reverberation contributions to form a portion of the spatial audio signal.

11. The apparatus according to claim 10, wherein the means configured to set reverberation decay to provide the defined contribution to the second acoustic environment, where the listener position is located within the region of the first acoustic environment defined by the threshold, when subject to claim 7, is configured to set reverberation decay to provide the defined contribution to the second acoustic environment based on the first function.

12. The apparatus according to any one of claims 1 to 11, wherein the means is configured to acquire a first distance threshold that at least partially defines an audio transition region that enables adaptive rendering between the first and second acoustic environments depending on the listening position in the audio scene, the means is configured to acquire at least one of the following: information indicating the number of acoustic environments to which the first distance threshold is applicable; information indicating which acoustic environment the first distance threshold is applicable to; information indicating which acoustic environment the first distance threshold is located in; information indicating a distance value relative to the first distance threshold; information indicating a shape or profile associated with the first distance threshold; information indicating a hysteresis offset value that defines a hysteresis region; information indicating an attenuation modulation type that defines a function for correcting the attenuation level when the listener position moves from outside the first distance threshold to the boundary between the first acoustic environment and the second acoustic environment; information indicating a hop that defines the number of hops from the current acoustic environment to the target acoustic environment; and information indicating a parameter attenuation upper limit that defines the maximum attenuation level.

13. An apparatus for generating an acoustic environment to assist in rendering an audio scene, the apparatus comprising means configured to acquire information of at least a first acoustic environment relating to the audio scene, wherein the audio scene comprises the first acoustic environment and a second environment, and to acquire a first distance threshold that at least partially defines an audio transition region enabling adaptive rendering between the first and second acoustic environments depending on the listening position within the audio scene, and to encode and output the information and the first distance threshold.

14. The apparatus according to claim 12, wherein the means is further configured to acquire a first function related to the first distance threshold, the first function defining an environmental modification characteristic of at least one of the first and second acoustic environments.

15. The apparatus according to claim 13 or 14, wherein the means is configured to acquire a first distance threshold that at least partially defines an audio transition region that enables adaptive rendering between the first and second acoustic environments depending on the listening position in the audio scene, the means is configured to acquire at least one of the following: information indicating the number of acoustic environments to which the first distance threshold is applicable; information indicating which acoustic environment the first distance threshold is applicable to; information indicating which acoustic environment the first distance threshold is located in; information indicating a distance value relative to the first distance threshold; information indicating a shape or profile associated with the first distance threshold; information indicating a hysteresis offset value that defines a hysteresis region; information indicating an attenuation modulation type that defines a function for correcting the attenuation level when the listener position moves from outside the first distance threshold to the boundary between the first and second acoustic environments; information indicating a hop that defines the number of hops from the current acoustic environment to the target acoustic environment; and information indicating a parameter attenuation upper limit that defines the maximum attenuation level.

16. The apparatus according to any one of claims 13 to 15, wherein the means configured to encode and output the aforementioned information is configured to generate a bitstream comprising encoded information and a first distance threshold.

17. The apparatus according to claim 16, wherein the means is further configured to acquire at least one audio signal associated with the audio scene, and the means is configured to encode and output the information, thereby generating the bitstream comprising the encoded at least one audio signal.

18. A method for a device for enabling audio transitions between at least two acoustic environments, the method comprising: acquiring information about at least a first acoustic environment associated with an audio scene, wherein the audio scene comprises a first acoustic environment and a second acoustic environment; acquiring a first distance threshold that at least partially defines an audio transition region enabling adaptive rendering between the first acoustic environment and the second acoustic environment, depending on the listening position within the audio scene; determining the listening position to adjust the environmental characteristics of at least one of the first and second acoustic environments; and adjusting the environmental characteristics of at least one of the first and second acoustic environments, depending on the listening position, wherein the environmental characteristics are adaptively controlled within the audio scene.

19. A method for a device for generating acoustic environment information to assist in rendering an audio scene, the method comprising: obtaining information of at least a first acoustic environment associated with an audio scene, wherein the audio scene comprises the first acoustic environment and a second environment; obtaining a first distance threshold that at least partially defines an audio transition region enabling adaptive rendering between the first acoustic environment and the second acoustic environment depending on the listening position in the audio scene; and encoding and outputting the information and the first distance threshold.

Images