Method, apparatus and system for three degrees of freedom (3dof+) extension of mpeg-h 3D audio

Abstract

To provide a method of processing position information indicative of an object position of an audio object.SOLUTION: An object position is usable for rendering of an audio object, which comprises: obtaining listener orientation information indicative of an orientation of the listener's head; obtaining listener displacement information indicative of displacement of the listener's head; determining the object position from the position information; modifying the object position on the basis of the listener displacement information by applying translation to the object position; and further modifying the modified object position on the basis of the listener orientation information. Further described is a corresponding apparatus for processing position information indicative of an object position of an audio object, wherein the object position is usable for rendering of the audio object.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to the following priority applications: U.S. Provisional Application No. 62 / 654,915 (Ref. No. D18045USP1), filed April 9, 2018; U.S. Provisional Application No. 62 / 695,446 (Ref. No. D18045USP2), filed July 9, 2018; and U.S. Provisional Application No. 62 / 823,159 (Ref. No. D18045USP3), filed March 25, 2019. These applications are incorporated herein by reference.

[0002] Technical Field The present disclosure relates to methods and apparatus for processing position information indicative of audio object positions and information indicative of a positional displacement of a listener's head. [Background technology]

[0003] The ISO / IEC23008-3 MPEG-H 3D Audio standard, version 1 (October 15, 2015) and amendments 1 to 4 do not provide for allowing for small translational movements of the user's head in a Three Degrees of Freedom (3DoF) environment. Summary of the Invention [Means for solving the problem]

[0004] The ISO / IEC23008-3 MPEG-H 3D Audio standard, version 1 (October 15, 2015) and amendments 1 to 4 provide features for the possibility of 3DoF environments in which the user (listener) performs head rotation movements. However, such features only support, at best, the signaling and corresponding rendering of rotational scene displacements. This means that the audio scene can remain spatially stationary under changes in the listener's head orientation that correspond to 3DoF attributes. However, within the current MPEG-H 3D Audio ecosystem, there is no possibility to take into account small translational movements of the user's head.

[0005] Thus, there is a need for a method and apparatus for processing position information of audio objects that can take into account small translational movements of a user's head, potentially in conjunction with rotational movements of the user's head.

[0006] The present disclosure provides an apparatus and a system for processing location information having the features of the respective independent and dependent claims.

[0007] According to an aspect of the present disclosure, a method for processing position information indicative of a position of an audio object is described. The processing may be compliant with the MPEG-H 3D Audio standard. The object position may be usable for rendering of the audio object. The audio object may be included in object-based audio content along with its position information. The position information may be (part of) metadata of the audio object. The audio content (e.g., the audio object with its position information) may be conveyed in an encoded audio bitstream. The method may include receiving the audio content (e.g., the encoded audio bitstream). The method may include obtaining listener orientation information indicative of an orientation of a listener's head. The listener may be referred to as, for example, a user of an audio decoder that performs the method. The listener's head orientation (listener orientation) may be an orientation of the listener's head with respect to a nominal orientation. The method may further include obtaining listener displacement information indicative of a displacement of the listener's head. The listener's head displacement may be a displacement with respect to a nominal listening position. The nominal listening position (or nominal listener position) may be a default position (e.g., a predetermined position, an expected position for the listener's head, or a sweet spot of a speaker arrangement). The listener orientation information and the listener displacement information may be obtained via an MPEG-H 3D Audio decoder input interface. The listener orientation information and the listener displacement information may be derived based on sensor information. The combination of the orientation information and the position information may be referred to as posture information. The method may further include determining an object position from the position information. For example, the object position may be extracted from the position information. The determination (e.g., extraction) of the object position may further be based on information regarding a geometry of a speaker arrangement of one or more speakers in the listening environment. The object position may also be referred to as a channel position of the audio object.The method may further include modifying the object position based on the listener displacement information by applying a translation to the object position. Modifying the object position may involve correcting the object position for a displacement of the listener's head from a nominal listening position. In other words, modifying the object position may involve applying a position displacement correction to the object position. The method may further include further modifying the modified object position based on the listener orientation information, for example by applying a rotational transformation to the modified object position (e.g., a rotation about the listener's head or the nominal listening position). Further modifying said modified object position to render the audio object may involve a rotational audio scene displacement.

[0008] Configured as described above, the proposed method provides a more realistic listening experience, especially for audio objects located close to the listener's head. In addition to the three (rotational) degrees of freedom that are typically provided to the listener in a 3DoF environment, the proposed method can also take into account the translational movement of the listener's head. This allows the listener to approach a nearby audio object from different angles, even from the side. For example, the listener can listen to a "mosquito" audio object close to the listener's head from different angles by slightly moving the head in addition to rotating the head. As a result, the proposed method can enable an improved, more realistic, and immersive listening experience for the listener.

[0009] In some embodiments, modifying the object position and further modifying the modified object position may be performed such that the audio object, after being rendered on one or more real or virtual speakers according to the further modified object position, is psychoacoustically perceived by the listener as originating from a fixed position relative to the nominal listening position, regardless of the displacement of the listener's head from the nominal listening position or the orientation of the listener's head with respect to the nominal orientation. Thus, the audio object may be perceived to move relative to the listener's head when the listener's head undergoes a displacement from the nominal listening position. Similarly, the audio object may be perceived to rotate relative to the listener's head when the listener's head undergoes a change in orientation from the nominal orientation. The one or more speakers may be, for example, part of a headset or may be part of a speaker arrangement (e.g., a 2.1, 5.1, 7.1, etc. speaker arrangement).

[0010] In some embodiments, modifying the object position based on the listener displacement information may be performed by translating the object position by a vector that is positively correlated to the absolute value and negatively correlated to the direction of the listener's head displacement vector from the nominal listening position.

[0011] It ensures that nearby audio objects are perceived by the listener to move in a manner consistent with the listener's head movements, which contributes to a more realistic listening experience for those audio objects.

[0012] In some embodiments, the listener displacement information may indicate the listener's head displacement from the nominal listening position due to small positional displacements. For example, the absolute value of the displacement may be less than or equal to 0.5 m. The displacement may be expressed in Cartesian coordinates (e.g., x, y, z) or spherical coordinates (e.g., azimuth, elevation, radius).

[0013] In some embodiments, the listener displacement information may indicate a displacement of the listener's head from a nominal listening position that is achievable by the listener moving his / her upper body and / or head. Thus, the displacement may be achievable for the listener without moving the lower body. For example, said displacement of the listener's head may be achievable when the listener is sitting in a chair.

[0014] In some embodiments, the position information may include an indication of the distance of the audio object from a nominal listening position. The distance (radial) may be less than 0.5 m. For example, the distance may be less than 1 cm. Alternatively, the distance of the audio object from the nominal listening position may be set to a default value by the decoder.

[0015] In some embodiments, the listener orientation information may include information regarding the yaw, pitch, and roll of the listener's head. The yaw, pitch, and roll may be given relative to a nominal orientation (e.g., a reference orientation) of the listener's head.

[0016] In some embodiments, the listener displacement information may include information regarding the listener's head displacement from a nominal listening position expressed in Cartesian or spherical coordinates, whereby the displacement may be expressed in x, y, z coordinates for Cartesian coordinates, or azimuth, elevation, radial coordinates for spherical coordinates.

[0017] In some embodiments, the method may further include detecting the listener's head orientation by wearable and / or static equipment. Similarly, the method may further include detecting the listener's head displacement from a nominal listening position by wearable and / or static equipment. The wearable equipment may be, correspond to and / or include, for example, a headset or an augmented reality (AR) / virtual reality (VR) headset. The static equipment may be, correspond to and / or include, for example, a camera sensor. This allows obtaining accurate information about the listener's head displacement and / or orientation, thereby enabling realistic processing of nearby audio objects depending on the orientation and / or displacement.

[0018] In some embodiments, the method may further include rendering the audio object to one or more real or virtual speakers according to the modified object position, for example, the audio object may be rendered to a left or right speaker of a headset.

[0019] In some embodiments, the rendering may be performed based on a Head Related Transfer Function (HRTF) for the listener's head, taking into account sonic occlusion for small distances of audio objects from the listener's head, such that the rendering of nearby audio objects is perceived as much more realistic by the listener.

[0020] In some embodiments, the modified object positions may further be adjusted to the input format used by an MPEG-H 3D Audio renderer. In some embodiments, the rendering may be performed using an MPEG-H 3D Audio renderer. In some embodiments, the processing may be performed using an MPEG-H 3D Audio decoder. In some embodiments, the processing may be performed by a scene displacement unit of an MPEG-H 3D Audio decoder. Thus, the proposed method allows implementing a limited six degrees of freedom (6DoF) experience (i.e. 3DoF+) in the framework of the MPEG-H 3D Audio standard.

[0021] According to another aspect of the disclosure, a further method of processing position information indicative of an object position of an audio object is described. The object position may be usable for rendering of the audio object. The method may include obtaining listener displacement information indicative of a displacement of a listener's head. The method may further include determining the object position from the position information. The method may further include modifying the object position based on the listener displacement information by applying a translation to the object position.

[0022] Configured as described above, the proposed method provides a more realistic listening experience, especially for audio objects located close to the listener's head. By being able to take into account small translational movements of the listener's head, the proposed method allows the listener to approach nearby audio objects from different angles, even from the side. As a result, the proposed method can enable an improved, more realistic and immersive listening experience for the listener.

[0023] In some embodiments, modifying the object position based on the listener displacement information may be performed such that the audio object, after being rendered to one or more real or virtual speakers according to the modified object position, is psychoacoustically perceived by the listener as emanating from a fixed position relative to the nominal listening position, regardless of the displacement of the listener's head from the nominal listening position.

[0024] In some embodiments, modifying the object position based on the listener displacement information may be performed by translating the object position by a vector that is positively correlated to the absolute value and negatively correlated to the direction of the listener's head displacement vector from the nominal listening position.

[0025] According to another aspect of the disclosure, a further method of processing position information indicative of an object position of an audio object is described. The object position may be usable for rendering of the audio object. The method may include obtaining listener orientation information indicative of an orientation of a listener's head. The method may further include determining an object position from the position information. The method may further include modifying the object position based on the listener orientation information, for example by applying a rotation transformation to the object position (e.g., a rotation about the listener's head or a nominal listening position).

[0026] Configured as described above, the proposed method is able to provide a more realistic listening experience to the listener by taking into account the orientation of the listener's head.

[0027] In some embodiments, modifying the object position based on the listener orientation information may be performed such that the audio object, after being rendered to one or more real or virtual loudspeakers according to the modified object position, is psychoacoustically perceived by the listener as emanating from a fixed position relative to a nominal listening position, regardless of the orientation of the listener's head with respect to the nominal orientation.

[0028] According to another aspect of the disclosure, an apparatus is described for processing position information indicative of an object position of an audio object. The object position may be usable for rendering of the audio object. The apparatus may include a processor and a memory coupled to the processor. The processor may be adapted to obtain listener orientation information indicative of an orientation of a listener's head. The processor may further be adapted to obtain listener displacement information indicative of a displacement of the listener's head. The processor may further be adapted to determine the object position from the position information. The processor may further be adapted to modify the object position based on the listener displacement information by applying a translation to the object position. The processor may further be adapted to further modify the modified object position based on the listener orientation information, for example by applying a rotation transformation (e.g., a rotation about the listener's head or a nominal listening position) to the modified object position.

[0029] In some embodiments, the processor may be adapted to modify the object position and further modify the modified object position such that the audio object, after being rendered to one or more real or virtual loudspeakers according to the further modified object position, is psychoacoustically perceived by a listener as originating from a fixed position relative to the nominal listening position, regardless of the displacement of the listener's head from the nominal listening position or the orientation of the listener's head with respect to the nominal orientation.

[0030] In some embodiments, the processor may be adapted to modify the object position based on the listener displacement information by translating the object position by a vector that is positively correlated to the absolute value and negatively correlated to the direction of the listener's head displacement vector from the nominal listening position.

[0031] In some embodiments, the listener displacement information may indicate the listener's head displacement from the nominal listening position due to small positional variations.

[0032] In some embodiments, the listener displacement information may indicate the displacement of the listener's head from a nominal listening position that the listener can achieve by moving their upper body and / or head.

[0033] In some embodiments, the position information may include an indication of the distance of the audio object from a nominal listening position.

[0034] In some embodiments, the listener orientation information may include information regarding the yaw, pitch, and roll of the listener's head.

[0035] In some embodiments, the listener displacement information may include information regarding the listener's head displacement from a nominal listening position expressed in Cartesian or spherical coordinates.

[0036] In some embodiments, the device may further include wearable and / or static equipment for detecting the orientation of the listener's head. In some embodiments, the device may further include wearable and / or static equipment for detecting the displacement of the listener's head from a nominal listening position.

[0037] In some embodiments, the processor may be further adapted to render the audio objects to one or more real or virtual loudspeakers according to the modified object positions.

[0038] In some embodiments, the processor may be adapted to perform said rendering based on an HRTF for the listener's head and taking into account sonic occlusion for small distances of audio objects from the listener's head.

[0039] In some embodiments, the processor may be further adapted to adjust the modified object positions to an input format used by an MPEG-H 3D Audio renderer. In some embodiments, the rendering may be performed using an MPEG-H 3D Audio renderer. In some embodiments, the processor may be adapted to implement an MPEG-H 3D Audio decoder. In some embodiments, the processor may be adapted to implement a scene displacement unit of an MPEG-H 3D Audio decoder.

[0040] According to another aspect of the disclosure, a further apparatus is described for processing position information indicative of an object position of an audio object. The object position may be usable for rendering of the audio object. The apparatus may include a processor and a memory coupled to the processor. The processor may be adapted to obtain listener displacement information indicative of a displacement of a listener's head. The processor may be further adapted to determine the object position from the position information. The processor may be further adapted to modify the object position based on the listener displacement information by applying a translation to the object position.

[0041] In some embodiments, the processor may be adapted to modify the object position based on the listener displacement information such that the audio object, after being rendered to one or more real or virtual loudspeakers according to the modified object position, is psychoacoustically perceived by the listener as emanating from a fixed position relative to the nominal listening position, regardless of a displacement of the listener's head from the nominal listening position.

[0042] In some embodiments, the processor may be adapted to modify the object position based on the listener displacement information by translating the object position by a vector that is positively correlated to the absolute value and negatively correlated to the direction of the listener's head displacement vector from the nominal listening position.

[0043] According to another aspect of the disclosure, a further apparatus is described for processing position information indicative of an object position of an audio object. The object position may be usable for rendering of the audio object. The apparatus may include a processor and a memory coupled to the processor. The processor may be adapted to obtain listener orientation information indicative of an orientation of a listener's head. The processor may be further adapted to determine the object position from the position information. The processor may be further adapted to modify the object position based on the listener orientation information, for example by applying a rotation transformation (e.g., a rotation about the listener's head or a nominal listening position) to the modified object position.

[0044] In some embodiments, the processor may be adapted to modify the object position based on the listener orientation information such that the audio object, after being rendered to one or more real or virtual loudspeakers according to the modified object position, is psychoacoustically perceived by the listener as emanating from a fixed position relative to a nominal listening position, regardless of the orientation of the listener's head with respect to the nominal orientation.

[0045] According to yet another aspect, a system is described that may include an apparatus according to any of the above aspects and wearable and / or static equipment capable of detecting a listener's head orientation and detecting a listener's head displacement.

[0046] It will be understood that the method steps and apparatus features may be interchanged in many ways. In particular, details of the disclosed methods may be implemented as an apparatus adapted to perform part or all of the method or steps, and vice versa, as will be understood by those skilled in the art. In particular, the apparatus according to the present disclosure may relate to an apparatus for implementing or performing the method according to the above embodiments and variations thereof, and it is understood that each statement made with respect to the method applies equally to the corresponding apparatus. Similarly, the method according to the present disclosure may relate to the operation method of the apparatus according to the above embodiments and variations thereof, and it is understood that each statement made with respect to the apparatus applies equally to the corresponding method. [Brief description of the drawings]

[0047] The invention is described below, by way of example, with reference to the accompanying drawings, in which: [Figure 1] 1 illustrates a schematic diagram of an example MPEG-H 3D Audio system. [Diagram 2] 1 illustrates a schematic diagram of an example MPEG-H 3D Audio system according to the present invention; [Diagram 3]1 illustrates a schematic diagram of an example of an audio rendering system according to the present invention; [Figure 4] 1 illustrates a schematic diagram of an exemplary set of Cartesian coordinate axes and their relationship to spherical coordinates. [Diagram 5] 3 is a flow chart that illustrates generally an example of a method for processing position information for audio objects according to the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0048] As used herein, 3DoF is a system that can correctly handle a user's head movements, especially head rotation, typically specified in three parameters (e.g., yaw, pitch, and roll). Such systems are often available in various gaming systems, such as virtual reality (VR) / augmented reality (AR) / mixed reality (MR) systems, or in other acoustic environments of that type.

[0049] As used herein, a user (eg, a user of an audio decoder or a playback system that includes an audio decoder) may also be referred to as a "listener."

[0050] As used herein, 3DoF+ means that in addition to the user's head movements that can be properly handled by a 3DoF system, small translational movements can also be handled.

[0051] As used herein, "small" refers to movement being limited to less than a threshold, typically 0.5 meters. This means that the movement is 0.5 meters or less from the user's original head position. For example, the user's movement is constrained by the user being seated in a chair.

[0052] As used herein, "MPEG-H 3D Audio" refers to the specifications standardized in ISO / IEC 23008-3 and / or any future amendments, editions or other versions of the ISO / IEC 23008-3 standard.

[0053] In the context of audio standards provided by the MPEG organization, the distinction between 3DoF and 3DoF+ can be defined as follows: · 3DoF: allows the user to experience yaw, pitch and roll movements (for example of the user's head); · 3DoF+: The user can experience yaw, pitch, roll movement (for example of the user's head) as well as limited translational movement, such as when sitting in a chair.

[0054] The limited (small) head translational movement may be a movement constrained to a certain radius of movement. For example, the movement may be constrained because the user is in a seated position, e.g., without using the lower body. The small head translational movement may be related to or correspond to a displacement of the user's head relative to a nominal listening position. The nominal listening position (or nominal listener position) may be a default position (e.g., a pre-determined position, an expected position of the listener's head, or a sweet spot of a speaker arrangement, etc.).

[0055] A 3DoF+ experience may be comparable to a constrained 6DoF experience, where translational motion can be described as limited or small head movements. In one example, audio is also rendered based on the position and orientation of the user's head, including possible sound masking. The rendering may be performed to take into account sound masking for small distances of audio objects from the listener's head, for example based on a Head Related Transfer Function (HRTF) for the listener's head.

[0056] For methods, systems, apparatus and other devices that conform to the functionality defined by the MPEG-H 3D Audio standard, it means that 3DoF+ will be enabled in any future versions of the MPEG standard, such as future versions of Omnidirectional Media Formats (e.g., as standardized in future versions of MPEG-I), and / or in any updates to MPEG-H Audio (e.g., amendments or newer standards based on the MPEG-H 3D Audio standard), or any other related or supporting standards that may require updates (e.g., standards that specify certain types of metadata and SEI messages).

[0057] For example, the audio renderer that is mandatory for the audio standard specified in the MPEG-H 3D Audio standard may be extended to include rendering of the audio scene, in order to accurately take into account the user interaction with the audio scene, for example when the user moves his / her head slightly sideways.

[0058] The present invention provides various technical advantages, including the advantage of providing MPEG-H 3D Audio that can handle 3DoF+ use cases. The present invention extends the MPEG-H 3D Audio standard to support 3DoF+ capabilities.

[0059] To support 3DoF+ functionality, the audio rendering system should take into account limited / small positional displacement of the user / listener's head. The positional displacement should be determined based on a relative offset from an initial position (i.e., default position / nominal listening position). In one example, the magnitude of this offset (e.g., P0 is the nominal listening position and P1 is the displaced position of the listener's head, r offset=||P0-P1||) has a maximum radial offset of about 0.5 m. In another example, the magnitude of the offset is limited to an offset that can be achieved while the user is sitting in a chair and performing no lower body movements (but the head is moving relative to the body). This (small) offset distance results in little (perceptual) level and pan difference for distant audio objects. However, for close objects even such a small offset distance can be perceptually significant. In fact, the listener's head movements can have a perceptual effect on the perception of where the correct audio object localization is. This perceptual effect can be caused by (i) the displacement of the user's head (e.g., r offset =||P0-P1||) and the distance to the audio object (e.g., r) remains significant as long as the ratio between the angle (e.g., r = ||P0-P1||) and the distance to the audio object (e.g., r) trigonometrically results in an angle that is within the range of the psychoacoustic ability of the user to detect the sound direction (i.e., perceptually detectable by the user / listener). Such range may vary for different audio renderer settings, audio material, and playback configurations. For example, if the localization accuracy range is, say, ±3° and the listener has a freedom of lateral movement of ±0.25m, this corresponds to an object distance of approximately 5m.

[0060] For objects close to the listener (e.g. objects less than 1m away from the user), it is important for 3DoF+ scenarios to properly handle position displacements of the listener's head, as these have large perceptual effects during both panning and level changes.

[0061] An example of handling objects close to the listener is when an audio object (e.g., a mosquito) is positioned very close to the listener's face. An audio system, such as an audio system providing VR / AR / MR functionality, should allow the user to perceive this audio object from all sides and angles, even while the user makes small translational head movements. For example, the user should be able to accurately perceive the object (e.g., a mosquito) even while the user moves his or her head without moving the lower body.

[0062] However, systems compatible with the current MPEG-H 3D Audio specification cannot handle this correctly. Instead, when using a system compatible with MPEG-H 3D Audio, the "mosquito" is perceived from the wrong position relative to the user. In scenarios involving 3DoF+ performance, small translational movements should produce a significant difference in the perception of the audio object (e.g., moving the head to the left, the "mosquito" audio object should be perceived from the right side relative to the user's head).

[0063] The MPEG-H 3D Audio standard includes a bitstream syntax that allows signaling of object distance information (starting from 0.5m) via the bitstream syntax, for example via the object_metadata() syntax element.

[0064] The syntax element prodMetadataConfig() may be introduced into the bitstream provided by the MPEG-H 3D Audio standard. It can be used to signal that the object distance is very close to the listener. For example, the syntax prodMetadataConfig() may signal that the distance between the user and the object is less than a certain threshold distance (e.g., <1 cm).

[0065] 1 and 2 show the invention based on headphone rendering (i.e. the loudspeakers move with the listener's head).

[0066] FIG. 1 shows an example of system behavior 100 compliant with the MPEG-H 3D Audio system. The example assumes that the listener's head is at position P0 103 at time t0 and moves to position P1 104 at time t1>t0. The dashed circle around positions P0 and P1 indicates the allowable 3DoF+ movement region (e.g., 0.5 m radius). Position A 101 indicates the signal object position (position at time t0 and time t1; i.e., the signaled object position is assumed to be constant in time). Position A also indicates the object position rendered by the MPEG-H 3D Audio renderer at time t0. Position B 102 indicates the object position rendered by MPEG-H 3D Audio at time t1. The vertical lines extending upward from positions P0 and P1 indicate the respective orientations (e.g., looking direction) of the listener's head at time t0 and time t1. The displacement of the user's head between positions P0 and P1 is r offset =||P0-P1|| 106. If the listener is located at the default position (nominal listening position) P0 103 at time t0, the listener will perceive the audio object (e.g., a mosquito) at the correct position A 101. If the user moves to position P1 104 at time t1, the MPEG-H 3D Audio processing is applied as currently standardized, resulting in an error δ AB If we introduce the error δ 105, the user will perceive the audio object at position B 102. That is, despite the listener's head movements, the audio object (e.g., a mosquito) will still be perceived as being located directly in front of the listener's head (i.e., substantially co-moving with the listener's head). Notably, the introduced error δ AB 105 occurs regardless of the listener's head orientation.

[0067] FIG. 2 shows an example of system behavior for an MPEG-H 3D Audio system 200 according to the present invention. In FIG. 2, the listener's head is located at position P0 203 at time t0 and moves to position P1 204 at time t1>t0. The dashed circle around positions P0 and P1 again indicates the allowable 3DoF+ movement region (e.g., 0.5m radius). In 201, it is shown that position A=B, i.e., the signaled object position (position at time t0 and time t1; i.e., the signaled object position is assumed to be constant over time). Position A=B 201 also shows the position of the object rendered by MPEG-H 3D Audio at time t0 and time t1. The vertical arrows extending upwards from positions P0 203 and P1 204 indicate the respective orientations (e.g., looking direction) of the listener's head at times t0 and t1. Since the listener is located at an initial / default position (nominal listening position) P0 203 at time t0, the listener perceives the audio object (e.g., a mosquito) at the correct position A 201. If the user moves to position P1 203 at time t1, under the present invention, the user will still perceive the audio object at a position B 201 that is similar (e.g., substantially equal) to position A 201. In this way, the present invention allows the user's position to change over time (e.g., from position P0 203 to position P1 204) while still perceiving the sound from the same (spatially fixed) position (e.g., position A=B 201, etc.). In other words, the audio object (e.g., a mosquito) moves relative to the listener's head in response to (e.g., negatively correlated with) the listener's head movement. This allows the user to move around the audio object (e.g., a mosquito) and perceive the audio object from different angles or sides. The displacement of the user's head between positions P0 and P1 is r offset =||P0-P1|| 206.

[0068] 3 shows an example of an audio rendering system 300 according to the invention. The audio rendering system 300 may correspond to or include a decoder, such as, for example, an MPEG-H 3D Audio decoder. The audio rendering system 300 may include an audio scene displacement unit 310 with a corresponding audio scene displacement processing interface (e.g., an interface for scene displacement data according to the MPEG-H 3D Audio standard). The audio scene displacement unit 310 may output object positions 321 for rendering the respective audio objects. For example, the scene displacement unit may output object position metadata for rendering the respective audio objects.

[0069] The audio rendering system 300 may further include an audio object renderer 320. For example, the renderer may consist of hardware, software, and / or any partial or complete process executed via cloud computing, including various services on the Internet, often referred to as the "cloud", such as software development platforms, servers, storage and software, compatible with the specifications defined by the MPEG-H 3D Audio standard. The audio object renderer 320 may render the audio objects to one or more (real or virtual) speakers according to the respective object positions (which may be modified or further modified object positions described below). The audio object renderer 320 may render the audio objects to headphones and / or loudspeakers. That is, the audio object renderer 320 may generate object waveforms according to a given playback format. To this end, the audio object renderer 320 may utilize compressed object metadata. Each object may be rendered into a certain output channel according to its object position (e.g., modified object position or further modified object position). The object positions may therefore be referred to as channel positions of their audio objects. The audio object positions 321 may be included in the object position metadata or scene displacement metadata output by the scene displacement unit 310.

[0070] The process of the present invention may be compliant with the MPEG-H 3D Audio standard. Thus, the process may be performed by an MPEG-H 3D Audio decoder, or more specifically, an MPEG-H scene displacement unit and / or an MPEG-H 3D Audio renderer. Thus, the audio rendering system 300 of FIG. 3 may correspond to or include an MPEG-H 3D Audio decoder (i.e., a decoder compliant with the specifications defined by the MPEG-H 3D Audio standard). In one example, the audio rendering system 300 may be a device having a processor and a memory coupled to the processor, the processor adapted to implement an MPEG-H 3D Audio decoder. In particular, the processor may be adapted to implement an MPEG-H scene displacement unit and / or an MPEG-H 3D Audio renderer. Thus, the processor may be adapted to perform the processing steps described in the present disclosure (e.g., steps S510 to S560 of the method 500 described below with reference to FIG. 5). In another example, the processing or audio rendering system 300 may be executed in the cloud.

[0071] The audio rendering system 300 may obtain (e.g., receive) listening position data 301. The audio rendering system 300 may obtain the listening position data 301 via an MPEG-H 3D Audio decoder input interface. The listening position data 301 may indicate the orientation and / or position (e.g., displacement) of a listener's head. Thus, the listening position data 301 (also referred to as posture information) may include listener orientation information and / or listener displacement information.

[0072] The listener displacement information may indicate the listener's head displacement (e.g., displacement from a nominal listening position). The listener displacement information may be represented by the absolute value r of the listener's head displacement from a nominal listening position, as shown in FIG. offset=||P0-P1|| 206 or may include an indication thereof. In the context of the present invention, the listener displacement information indicates a small positional displacement of the listener's head from the nominal listening position. For example, the absolute value of the displacement may be 0.5 m or less. Typically, this is a displacement of the listener's head from the nominal listening position that is achievable by the listener moving his / her upper body and / or head. That is, the displacement may be achievable for the listener without moving his / her lower body. For example, the listener's head displacement may be achievable when the listener is sitting in a chair, as described above. The displacement may be expressed in various coordinate systems, such as, for example, Cartesian coordinates (e.g., in terms of x, y, z) or spherical coordinates (e.g., in terms of azimuth, elevation, radius). It should be understood that alternative coordinate systems for expressing the listener's head displacement are also feasible and are encompassed by the present disclosure.

[0073] The listener orientation information may indicate an orientation of the listener's head (e.g., an orientation of the listener's head with respect to a nominal / reference orientation of the listener's head). For example, the listener orientation information may include information regarding the yaw, pitch, and roll of the listener's head, where the yaw, pitch, and roll may be given with respect to the nominal orientation.

[0074] The listening position data 301 may be continuously collected from a receiver, which may provide information about the translational movement of the user. For example, the listening position data 301 used at a given time may be the one recently collected from the receiver. The listening position data may be derived / collected / generated based on sensor information. For example, the listening position data 301 may be derived / collected / generated by wearable and / or static equipment having appropriate sensors. That is, the listener's head orientation may be detected by the wearable and / or static equipment. Similarly, the listener's head displacement (e.g., displacement from a nominal listening position) may be detected by the wearable and / or static equipment. The wearable equipment may be, correspond to, and / or include, for example, a headset (e.g., an AR / VR headset). The static equipment may be, correspond to, and / or include, for example, a camera sensor. The static equipment may be included in, for example, a TV set or a set-top box. In some embodiments, the listening position data 301 may be received from an audio encoder (e.g., an MPEG-H 3D Audio compliant encoder), which may have acquired (e.g., received) the sensor information.

[0075] In one example, wearable and / or static equipment for detecting listening position data 301 may be referred to as a tracking device that supports head position estimation / detection and / or head orientation estimation / detection. There are various solutions that allow accurate tracking of a user's head movements using a computer or smartphone camera (e.g., "FaceTrackNoIR", "opentrack" based on face recognition and tracking). Also, some head-mounted display (HMD) virtual reality systems (e.g., HTC VIVE, Oculus Rift) have integrated head tracking technology. Any of these solutions may be used in the context of the present disclosure.

[0076] It is also important to note that head displacement distances in the physical world do not necessarily correspond one-to-one to the displacements indicated by the listening position data 301. To achieve hyper-realistic effects (e.g., an over-amplified user motion parallax effect), certain applications may use different sensor calibration settings or specify different mappings between movements in real and virtual spaces. Thus, in some use cases, small physical movements can be expected to result in larger displacements in virtual reality. In any case, it can be said that the absolute values ​​of the displacements in the physical world and virtual reality (i.e., the displacements indicated by the listening position data 301) are positively correlated. Similarly, the directions of the displacements in the physical world and virtual reality are also positively correlated.

[0077] The audio rendering system 300 may further receive (object) position information (e.g., object position data) 302 and audio data 322. The audio data 322 may include one or more audio objects. The position information 302 may be part of the metadata of the audio data 322. The position information 302 may indicate an object position of each of the one or more audio objects. For example, the position information 302 may include an indication of a distance of each audio object relative to a nominal listening position of a user / listener. The distance (radial) may be less than 0.5 m. For example, the distance may be less than 1 cm. If the position information 302 does not include an indication of a distance of a given audio object from the nominal listening position, the audio rendering system may set the distance of this audio object from the nominal listening position to a default value (e.g., 1 m). The position information 302 may further include an indication of an elevation angle and / or an azimuth angle of the respective audio object.

[0078] Each object position may be usable to render a corresponding audio object. Thus, the position information 302 and the audio data 322 may be included in or may be in the form of object-based audio content. The audio content (e.g., the audio objects / audio data 322 and its position information 302) may be conveyed in an encoded audio bitstream. For example, the audio content may be in the format of a bitstream received from transmission over a network. In this case, the audio rendering system may be said to receive the audio content (e.g., from an encoded audio bitstream).

[0079] In one example of the present invention, metadata parameters may be used to correct the processing of certain use cases of backward compatible enhancements for 3DoF and 3DoF+. The metadata may include listener displacement information in addition to listener orientation information. Such metadata parameters may be utilized by the systems shown in Figures 2 and 3, as well as any other embodiments of the present invention.

[0080] Backward compatible enhancements may allow to correct the use case processing (e.g., implementation of the present invention) based on the normative MPEG-H 3D Audio scene displacement interface. This means that a legacy MPEG-H 3D Audio decoder / renderer will still generate output, even if incorrect. However, an enhanced MPEG-H 3D Audio decoder / renderer according to the present invention will correctly apply the extension data (e.g., extended metadata) and processing, and thus be able to handle scenarios of objects located close to the listener in a correct manner.

[0081] In one example, the invention relates to providing data for small translational movements of a user's head in a different format than that outlined below, and these formulas can be adapted accordingly. For example, the data may be provided in a format such as x, y, z coordinates (in a Cartesian coordinate system) instead of azimuth, elevation and radius (in a spherical coordinate system). An example of the relationship of these coordinate systems to each other is shown in Figure 4.

[0082] In one example, the present invention is directed to providing metadata for inputting translational motion of a listener's head (e.g., listener displacement information included in listening position data 301 shown in FIG. 3). The metadata may be used, for example, for interfacing for scene displacement data. The metadata (e.g., listener displacement information) may be obtained by deployment of a tracking device supporting 3DoF+ or 6DoF tracking.

[0083] In one example, metadata (e.g., listener displacement information, in particular listener head displacement, or equivalently, scene displacement) may be represented by three parameters sd_azimuth, sd_elevation, and sd_radius, which relate to the azimuth, elevation, and radius (spherical coordinates) of the listener's head displacement (or scene displacement).

[0084] The syntax for these parameters is given by the following table: [Table 1]

[0085] In another example, the metadata (e.g., listener displacement information) may be represented by the following three parameters sd_x, sd_y, and sd_z in Cartesian coordinates, which reduces the processing of the data from spherical coordinates to Cartesian coordinates. The metadata may be based on the following syntax: [Table 2] As mentioned above, the above syntax, or its equivalents, may signal information related to rotations around the x, y, and z axes.

[0086] In one example of the present invention, the processing of scene displacement angles for channels and objects may be improved by extending the equations to take into account the position change of the user's head, i.e., the processing of object positions may take into account listener displacement information (e.g., may be based at least in part on listener shift information).

[0087] An example of a method 500 for processing position information indicative of object positions of audio objects is shown in the flow chart of Figure 5. The method may be performed by a decoder, such as an MPEG-H 3D Audio decoder. The audio rendering system 300 of Figure 3 may be an example of such a decoder.

[0088] As a first step (not shown in FIG. 5), audio content including audio objects and corresponding position information is received, for example from an encoded audio bitstream. The method may then further include decoding the encoded audio content to obtain the audio objects and the position information.

[0089] Step S510 In, listener orientation information is obtained (e.g., received). The listener orientation information may indicate an orientation of a listener's head.

[0090] Step S520 In, listener displacement information is obtained (eg, received). The listener displacement information may indicate a listener's head displacement.

[0091] Step S530In, the object position is determined from the position information. For example, the object position (e.g., azimuth, elevation, radius, or x, y, z or their equivalents) may be extracted from the position information. The determination of the object position may also be based, at least in part, on information regarding the geometry of a speaker arrangement of one or more (real or virtual) speakers in the listening environment. If the radius is not included in the position information for that audio object, the decoder may set the radius to a default value (e.g., 1 m).

[0092] In some embodiments, the default value may depend on the geometry of the speaker arrangement.

[0093] In particular, steps S510, S520, and S520 may be performed in any order.

[0094] Step S540 In step S530, the object position determined is modified based on the listener displacement information. This may be done by applying a translation to the object position according to the displacement information (e.g., in response to the listener's head displacement). Thus, modifying the object position can be said to relate to correcting the object position for the listener's head displacement (e.g., displacement from a nominal listening position). In particular, modifying the object position based on the listener displacement information may be performed by translating the object position by a vector that is negatively correlated to the direction and positively correlated to the absolute value of the listener's head displacement vector from the nominal listener position. An example of such a translation is shown diagrammatically in FIG. 2.

[0095] Step S550Now, the modified object positions obtained in step S540 are further modified based on the listener orientation information. For example, this may be done by applying a rotation transformation to the modified object positions according to the listener orientation information. This rotation may be, for example, a rotation relative to the listener's head or a nominal listening position. The rotation transformation may be performed by a scene displacement algorithm.

[0096] As mentioned above, user offset compensation (i.e., modification of object position based on listener displacement information) is taken into account when applying the rotation transformation. For example, applying the rotation transformation may include: Calculation of the rotation transformation matrix (based on user orientation, e.g. listener orientation information), Transformation of object positions from spherical to Cartesian coordinates, Applying a rotation transformation to the audio object (i.e. to correct object position) with user position offset compensation, Converting the object position after a rotation transformation from Cartesian coordinates back to spherical coordinates.

[0097] Further Step S560 Method 500 may further include rendering the audio objects to one or more real or virtual speakers according to the modified object positions (not shown in FIG. 5).

[0098] To this end, the further modified object position may be adjusted to an input format used by an MPEG-H 3D Audio renderer (e.g., audio object renderer 320 described above). The aforementioned one or more (real or virtual) speakers may, for example, be part of a headset or may be part of a speaker arrangement (e.g., a 2.1 speaker arrangement, a 5.1 speaker arrangement, a 7.1 speaker arrangement, etc.). In some embodiments, the audio object may, for example, be rendered to the left and right speakers of a headset.

[0099] The aim of the above steps S540 and S550 is as follows: modifying the object position and further modifying the modified object position are performed such that the audio object, after being rendered to one or more (real or virtual) loudspeakers according to the further modified object position, is psychoacoustically perceived by the listener as originating from a fixed position relative to the nominal listening position. This fixed position of the audio object is psychoacoustically perceived regardless of the displacement of the listener's head from the nominal listening position and regardless of the orientation of the listener's head with respect to the nominal orientation. In other words, the audio object may be perceived to move (translate) relative to the listener's head when the listener's head undergoes a displacement from the nominal listening position. Similarly, the audio object may be perceived to move (rotate) relative to the listener's head when the listener's head undergoes a change in orientation from the nominal orientation. Thereby, the listener can perceive nearby audio objects from different angles and distances by moving his / her head.

[0100] The modification of the object positions and the further modification of the modified object positions in steps S540 and S550 may be performed in the context of (rotational / translational) audio scene displacement, for example by the audio scene displacement unit 310 described above.

[0101] It should be noted that depending on the particular use case being faced, certain steps may be omitted. For example, if the listening position data 301 includes only listener displacement information (but does not include listener orientation information or includes only listener orientation information indicating that the listener's head orientation has not deviated from the nominal orientation), step S550 may be omitted. Then, the rendering in step S560 is performed according to the modified object position determined in step S540. Similarly, if the listener position information 301 includes only listener orientation information (but does not include listener displacement information or includes only listener displacement information indicating that the listener's head position has not deviated from the nominal listener position), step S540 may be omitted. Then, step S550 concerns modifying the object position determined in step S530 based on the listener orientation information. Then, the rendering in step S560 is performed according to the modified object position determined in step S550.

[0102] Broadly speaking, the present invention proposes position updating of object positions (e.g., position information 302 accompanying audio data 322) received as part of object-based audio content based on listening position data 301 for a listener.

[0103] First, the object position (or channel position) p(az,el,r) is determined. This may be performed in the context of (e.g., as part of) step 530 of method 500.

[0104] For channel-based signals, the radial r may be determined as follows: If the intended loudspeaker (for the channel of the channel-based input signal) is present in a playback loudspeaker setup and the distance of the playback setup is known, the radius r is set to the loudspeaker distance (e.g. in cm). If the intended loudspeaker is not present in the playback speaker setup, but the distance of the playback speakers is known (e.g. from the nominal listening position), then radius r is set to the maximum playback speaker distance. If the intended loudspeaker is not present in the playback speaker setup and there is no known playback speaker distance, the radius r is set to a default value (e.g. 1023 cm).

[0105] For object-based signals, the radial coordinate r is determined as follows: If the object distance is known (e.g. communicated in prodMetadataConfig() from the production tools and production format), the radial r is set to the known object distance (e.g. signaled by goa_bsObjectDistance[] (in cm) according to table AMD5.7 of the MPEG-H 3D Audio standard). [Table 3] If the object distance is known from the position information (e.g., from object metadata, conveyed in object_metadata()), then radius r is set to the object distance signaled in the position information (e.g., radius[] in cm conveyed in the object metadata). Radius r may be signaled according to the "Object Metadata Scaling" and "Object Metadata Constraints" sections below.

[0106] Scaling object metadata As an optional step in the context of determining the object position, the object position p=(az,el,r) determined from the position information may be scaled. This may involve applying a scaling factor to invert the encoder scaling of the input data for each component. This may be performed for all objects. The actual scaling of the object position may be implemented along the lines of the following pseudocode: [Table 4]

[0107] Object Metadata Restrictions As a further optional step in the context of determining the object position p=(az,el,r), the (possibly scaled) object position determined from the position information may be constrained. This may involve applying constraints to the decoded values ​​for each component in order to keep the values ​​within a valid range. This may be performed for all objects. The actual constraints on the object positions may be implemented according to the following pseudocode function: [Table 5]

[0108] The determined (and optionally scaled and / or constrained) object position may then be transformed into a given coordinate system, for example a "common convention" coordinate system where azimuth 0° is at the right ear (positive values ​​counterclockwise) and elevation 0° is at the top of the head (positive values ​​downwards). Thus, the object position p may be transformed into a position according to the "common" convention. This gives an object position p' such that: p'(az',el',r) az'=az+90° el'=90°-el The radius r does not change.

[0109] At the same time, the listener displacement information (az offset ,el offset ,r offset ) may be transformed into a given coordinate system. Using "common convention", this becomes:

[0110] a-z' offset =az offset +90° el' offset =90°-el offset The radius r does not change.

[0111] In particular, the transformation into said predefined coordinate system for both object positions and listener head displacements may be performed in the context of step S530 or step S540.

[0112] The actual location update may be performed in the context of (eg, as part of) step S540 of method 500. The location update may include the following steps.

[0113] As a first step, the position p, or the position p' if a transformation to the given coordinate system has been performed, is transformed into Cartesian coordinates (x,y,z). In the following, without any intention of limitation, the process is described for the position p' in the given coordinate system. Also, without any intention of limitation, the following orientation / direction of the coordinate axes may be assumed: x-axis points to the right (as viewed from the listener's head when in the nominal orientation), y-axis points straight ahead, and z-axis points straight up. At the same time, the displacement information of the listener (az' offset ,el' offset ,r offset ) is transformed into Cartesian coordinates.

[0114] As a second step, the object position in Cartesian coordinates is shifted (translated) according to the listener's head displacement (scene displacement) in the manner described above. This may proceed as follows:

number

[0115] The shifted object position in Cartesian coordinates may be transformed into spherical coordinates and referred to as p". Following common convention, the shifted object position can be expressed in a given coordinate system as p"=(az",el",r).

[0116] Small resulting in a change in radial parameters

number

[0117] In another example, this can result in significant radial parameter changes (i.e., r'≫r). Large In the presence of a listener's head displacement, the modified position p" of the object can also be defined with a modified radial parameter r' as p"=(az",el",r') instead of p"=(az",el",r).

[0118] The corresponding value of the modified radial parameter r' is determined by the listener's head displacement distance (i.e., r offset =||P0-P1||) and the initial radial parameter (i.e., r=||P0-A||) (see, for example, Figures 1 and 2). For example, the corrected radial parameter r' can be determined based on the following trigonometric relationship:

number

[0119] This mapping of modified radial parameter r' to object / channel gains and its subsequent application for audio rendering can significantly improve the perceptual effect of level changes due to user movement. Allowing such modification of radial parameter r' enables an "adaptive sweet spot", meaning that the MPEG rendering system dynamically adjusts the sweet spot position depending on the listener's current position. In general, the rendering of audio objects according to modified (or further modified) object positions may be based on the modified radial parameter r'. In particular, the object / channel gains for rendering the audio objects may be based on (e.g., modified based on) the modified radial parameter r'.

[0120] Another example is during loudspeaker playback setup and rendering (e.g. Step S560 above In some cases, scene displacement can be disabled. However, optional enabling of scene displacement may be available, allowing the 3DoF+ renderer to generate a dynamically adjustable sweet spot depending on the listener's current position and orientation.

[0121] In particular, the step of transforming object positions and listener head displacement into Cartesian coordinates is optional, and the translation / shift (correction) according to listener head displacement (scene displacement) can be performed in any suitable coordinate system, in other words, the choice of Cartesian coordinates above should be understood as a non-limiting example.

[0122] In some embodiments, the scene displacement process (including modification of object positions and / or further modification of modified object positions) is controlled by flags (fields, elements, set bits) in the bitstream (e.g., useTrackingModeThe subclauses "17.3 Interface for local loudspeaker setup and rendering" and "17.4 Interface for binaural room impulse response (BRIR)" in ISO / IEC 23008-3 activate scene displacement processing. useTrackingMode In the context of this disclosure, useTrackingMode The element defines whether processing of scene displacement values ​​transmitted via the mpegh3daSceneDisplacementData() and mpegh3daPositionalSceneDisplacementData() interfaces is performed (subclause 17.3). Alternatively or additionally (subclause 17.4): useTrackingMode The field defines if a tracker device is connected and binaural rendering is processed in a special head tracking mode, which means processing of scene displacement values ​​sent via the mpegh3daSceneDisplacementData() and mpegh3daPositionalSceneDisplacementData() interfaces is performed.

[0123] The methods and systems described herein may be implemented as software, firmware and / or hardware. Certain components may be implemented as software running on, for example, a digital signal processor or a microprocessor. Other components may be implemented as hardware and / or as an application specific integrated circuit. Signals encountered in the methods and systems described above may be stored in media such as random access memory or optical storage media. The signals may be transferred over a network such as an airwave network, a satellite network, a wireless network, or a wired network such as the Internet. A typical device utilizing the methods and systems described herein is a portable electronic device or other consumer equipment used to store and / or render audio signals.

[0124] Although this document refers to MPEG, and in particular MPEG-H 3D Audio, this disclosure should not be construed as limited to these standards. Rather, as will be appreciated by those skilled in the art, this disclosure may find advantageous application in other standards for audio coding. Furthermore, although this document frequently refers to small positional displacements of the listener's head (e.g., from a nominal listening position), this disclosure is not limited to small positional displacements, but is generally applicable to any positional displacements of the listener's head.

[0125] It should be noted that the specification and drawings merely illustrate the principles of the proposed method, system and device. Those skilled in the art will be able to implement various configurations that embody the principles of the present invention and are within its spirit and scope, although not explicitly described or shown herein. Furthermore, all examples and embodiments outlined in this document are expressly intended primarily for illustrative purposes only to aid the reader in understanding the principles of the proposed method. Furthermore, all statements in this document providing principles, aspects and embodiments of the present invention, as well as specific examples thereof, are intended to encompass their equivalents.

[0126] In addition to the above, various example implementations and example embodiments of the present invention will become apparent from the following enumerated example embodiments (EEE), which are not claims.

[0127] A first EEE relates to a method for decoding an encoded audio signal bitstream, the method comprising the steps of: receiving, by an audio decoding device 300, an encoded audio signal bitstream (302, 322), the encoded audio signal bitstream including encoded audio data (322) and metadata corresponding to at least one object audio signal (302); decoding, by the audio decoding device (300), the encoded audio signal bitstream (302, 322) to obtain a representation of a plurality of sound sources; receiving, by the audio decoding device (300), listening position data (301); and generating, by the audio decoding device (300), audio object position data (321), the audio object position data (321) describing a plurality of sound sources relative to a listening position based on the listening position data (301).

[0128] The second EEE relates to a method of the first EEE, wherein said listening position data (301) is based on a first set of first translational position data and a second set of second translational position and orientation data.

[0129] A third EEE relates to the method of the second EEE, wherein either the first translational position data or the second translational position data is based on at least one of a set of spherical coordinates or a set of Cartesian coordinates.

[0130] The fourth EEE relates to the method of the first EEE, in which the listening position data (301) is obtained via an MPEG-H 3D Audio decoder input interface. A fifth EEE relates to the method of the first EEE, wherein the encoded audio signal bitstream includes an MPEG-H 3D Audio bitstream syntax element, the MPEG-H 3D Audio bitstream syntax element including encoded audio data (322) and metadata (302) corresponding to at least one object audio signal.

[0131] A sixth EEE relates to the method of the first EEE, further comprising the step of rendering, by the audio decoding device (300), the multiple sound sources to multiple loudspeakers, the rendering process being compliant with at least the MPEG-H 3D Audio standard.

[0132] A seventh EEE relates to the method of the first EEE, further comprising a step of transforming, by the audio decoding device (300), a position (302) corresponding to the at least one object audio signal into a second position (321) corresponding to an audio object position based on a translation of listening position data (301).

[0133] The eighth aspect of the present invention relates to the method of the seventh aspect, wherein the position p' of the audio object positions is expressed in a predetermined coordinate system (e.g., according to common convention) as p' = (a-z', e-l', r) az'=az+90° el'=90°-el a-z' offset =az offset +90° el' offset =90°-el offset is determined based on where az corresponds to a first azimuth parameter, el corresponds to a first elevation parameter, and r corresponds to a first radial parameter, where az' corresponds to a second azimuth parameter, el' corresponds to a second elevation parameter, and r' corresponds to a second radial parameter, and az offset corresponds to the third azimuthal parameter, and el offset corresponds to the third elevation parameter, and az offset corresponds to the fourth azimuthal parameter, and el offset corresponds to the fourth elevation parameter.

[0134] The 9th EEE relates to the method of the 8th EEE, where the shifted audio object position (321) of the audio object position (302) is given in Cartesian coordinates (x, y, z):

number

[0135] The tenth aspect of the present invention relates to the method of the ninth aspect of the present invention, wherein the parameter x offset ,y offset ,z offset teeth,

number

[0136] The eleventh EEE relates to the method of the seventh EEE, and is offset is for the scene displacement azimuth position,

number

number

number

[0137] The twelfth EEE relates to the method of the tenth EEE, wherein x offset The parameters relate the scene displacement offset position sd_x to the x-axis direction; offset The parameters relate the scene displacement offset position sd_y to the y-axis direction; z offset The parameter relates the scene displacement offset position sd_z to the direction of the z axis.

[0138] A thirteenth EEE relates to the method of the first EEE, further comprising a step of interpolating, by the audio decoding device, the first position data related to the listening position data (301) and the object audio signal (102) at an update rate.

[0139] A fourteenth EEE relates to the method of the first EEE, further comprising determining, by said audio decoding device 300, an efficient entropy coding of the listening position data (301).

[0140] A fifteenth EEE relates to the method of the first EEE, wherein the location data (301) regarding the listening position is derived based on sensor information.

Claims

1. 1. A method of processing position information indicative of an object position of an audio object, the processing being performed using an MPEG-H 3D Audio decoder, the object position being usable for rendering of the audio object, the method comprising: obtaining listener orientation information indicative of an orientation of a listener's head, the listener orientation information including information regarding yaw, pitch, and roll of the listener's head; obtaining listener displacement information indicative of a displacement of the listener's head relative to a nominal listening position via an MPEG-H 3D Audio decoder input interface; determining the object position from the position information; modifying the object position based on the listener displacement information by applying a translation to the object position; Further modifying the modified object position based on the listener orientation information, when the listener displacement information indicates a displacement of the listener's head from the nominal listening position due to a small positional displacement, and the small positional displacement has an absolute value of 0.5 meters or less, a distance between the audio object position and the listening position after the listener's head displacement is equal to a distance between the modified audio object position and the nominal listening position; and rendering said audio objects to one or more real or virtual loudspeakers according to said further modified object positions. method.

2. 2. The method of claim 1, wherein modifying the object position and further modifying the modified object position is performed such that the audio object, after being rendered to one or more real or virtual loudspeakers according to the further modified object position, is psychoacoustically perceived by a listener as emanating from a fixed position relative to a nominal listening position, regardless of a displacement of the listener's head from a nominal listening position or an orientation of the listener's head with respect to the nominal orientation.

3. The method of claim 1 , wherein modifying an object position based on the listener displacement information is performed by translating the object position for an equal but opposite displacement of the listener's head from a nominal listening position.

4. The method of claim 1 , wherein the listener displacement information indicates a listener's head displacement from a nominal listening position that can be achieved by the listener moving at least one of his or her upper body and / or head.

5. The method of claim 1 , further comprising detecting a listener's head orientation by wearable and / or static equipment.

6. 2. The method of claim 1, comprising detecting the displacement of the listener's head from a nominal listening position by wearable and / or static equipment.

7. The method of claim 1 , wherein the distance between the modified audio object position and the displaced listening position is mapped to a gain for audio level modification.

8. 13. A non-transitory computer readable medium having instructions which, when executed by a digital signal processor or microprocessor, cause the digital signal processor or microprocessor to perform the method of claim 1.

9. 1. An MPEG-H 3D Audio decoder for processing position information indicative of an object position of an audio object, the object position usable for rendering the audio object, the decoder comprising a processor and a memory coupled to the processor, the processor comprising: obtaining listener orientation information indicative of an orientation of a listener's head, the listener orientation information including information regarding yaw, pitch, and roll of the listener's head; obtaining listener displacement information indicative of a displacement of the listener's head relative to a nominal listening position via an MPEG-H 3D Audio decoder input interface; determining the object position from the position information; modifying the object position based on the listener displacement information by applying a translation to the object position; further modifying the modified object position based on the listener orientation information, when the listener displacement information indicates a displacement of the listener's head from the nominal listening position due to a small positional displacement, and the small positional displacement has an absolute value of 0.5 meters or less, a distance between the audio object position and the listening position after the listener's head displacement is equal to a distance between the modified audio object position and the nominal listening position; and rendering said audio objects onto one or more real or virtual loudspeakers according to said further modified object positions. decoder.

Images