Method, apparatus and system for three degree of freedom (3DOF +) extension of MPEG-H 3D audio

Abstract

The invention relates to a method, a device and a system for three degree of freedom (3DOF +) extension of MPEG-H 3D audio. There is described a method of processing position information indicative of an object position of an audio object, where the object position is usable for rendering the audio object, the method comprising: obtaining listener orientation information indicative of an orientation of a listener's head; obtaining listener displacement information indicative of a displacement of the listener's head; determining the position of the object according to the position information; modifying the object position based on the listener displacement information by applying translation to the object position; and further modifying the modified object position based on the listener orientation information. Further described is a corresponding device for processing location information indicative of an object location of an audio object, wherein the object location is usable to render the audio object.

Description

[0001] Information about divisional applications

[0002] This case is a divisional application. The parent case of this divisional application is an invention patent application with a filing date of April 9, 2019, application number 202111293975.4, and invention name "Method, device and system for three degrees of freedom (3DOF+) extension for MPEG-H 3D audio"; the parent case is a divisional application of an invention patent application with a filing date of April 9, 2019, application number 201980018139.X, and invention name "Method, device and system for three degrees of freedom (3DOF+) extension for MPEG-H 3D audio".

[0003] Cross-reference to related applications

[0004] This application claims priority to the following priority applications: U.S. Provisional Application 62 / 654,915 filed on April 9, 2018 (reference: D18045USP1); U.S. Provisional Application 62 / 695,446 filed on July 9, 2018 (reference: D18045USP2) and U.S. Provisional Application 62 / 823,159 filed on March 25, 2019 (reference: D18045USP3), which are incorporated herein by reference. Technical Field

[0005] The present disclosure relates to methods and apparatus for processing position information indicating the position of an audio object and information indicating the displacement of a listener's head position. Background Art

[0006] The first version (October 15, 2015) and Amendments 1-4 of the ISO / IEC 23008-3 MPEG-H 3D Audio standard do not provide for allowing certain small translational movements of the user's head in a three-degree-of-freedom (3DoF) environment. Summary of the invention

[0007] The first version (October 15, 2015) and Amendments 1-4 of the ISO / IEC 23008-3 MPEG-H 3D Audio standard provide functionality for the possibility of 3DoF environments, where the user (listener) performs head rotation movements. However, such functionality at best only supports rotational scene displacement signaling and corresponding rendering. This means that the audio scene can remain spatially fixed in case of changes in the listener's head orientation, which corresponds to the 3DoF property. However, within the current MPEG-H 3D Audio ecosystem, it is not possible to take into account certain small translational movements of the user's head.

[0008] Therefore, there is a need for methods and apparatus for processing position information of audio objects that can account for certain small translational movements of a user's head, potentially in conjunction with rotational movements of the user's head.

[0009] The present disclosure provides an apparatus and a system for processing location information, the apparatus and the system having the features of the respective independent and dependent claims.

[0010] According to one aspect of the present disclosure, a method for processing position information indicating a position of an audio object is described, wherein the processing may comply with the MPEG-H 3D audio standard. The object position may be used to render the audio object. The audio object may be included in object-based audio content together with its position information. The position information may be (part of) metadata of the audio object. The audio content (e.g., the audio object and its position information) may be transmitted in an encoded audio bitstream. The method may include receiving the audio content (e.g., the encoded audio bitstream). The method may include obtaining listener orientation information indicating an orientation of a listener's head. The listener may be referred to as a user (e.g., of an audio decoder performing the method). The orientation of the listener's head (listener orientation) may be an orientation of the listener's head relative to a nominal orientation. The method may further include obtaining listener displacement information indicating a displacement of the listener's head. The displacement of the listener's head may be a displacement relative to a nominal listening position. The nominal listening position (or nominal listener position) may be a default position (e.g., a predetermined position, an expected position of the listener's head, or an optimal point of a speaker arrangement). The listener orientation information and the listener displacement information may be obtained through the MPEG-H 3D audio decoder input interface. The listener orientation information and the listener displacement information may be derived based on the sensor information. The combination of the orientation information and the position information may be referred to as posture information. The method may further include determining the object position according to the position information. For example, the object position may be extracted from the position information. The determination (e.g., extraction) of the object position may be further based on information about the geometry of the speaker arrangement of one or more speakers in the listening environment. The object position may also be referred to as the channel position of the audio object. The method may further include modifying the object position based on the listener displacement information by applying a translation to the object position. Modifying the object position may involve correcting the object position for the displacement of the listener head from the nominal listening position. In other words, modifying the object position may involve applying a position displacement compensation to the object position. The method may further include, for example, further modifying the modified object position based on the listener orientation information by applying a rotation transformation (e.g., a rotation relative to the listener head or the nominal listening position) to the modified object position. Further modifying the modified object position for rendering the audio object may involve rotating the audio scene displacement.

[0011] Configured as described above, the proposed method provides a more realistic listening experience, especially for audio objects positioned close to the listener's head. In addition to the three (rotational) degrees of freedom conventionally provided to the listener in a 3DoF environment, the proposed method can also take into account translational movements of the listener's head. This enables the listener to approach close audio objects from different angles and even from the side. For example, a listener may be able to listen to a "mosquito" audio object close to the listener's head from different angles by slightly moving his head in addition to rotating his head. Therefore, the proposed method can achieve an improved, more realistic immersive listening experience for the listener.

[0012] In some embodiments, modifying the object position and further modifying the modified object position can be performed such that after rendering to one or more real speakers or virtual speakers according to the further modified object position, the audio object is psychoacoustically perceived by the listener as originating from a position fixed relative to a nominal listening position, regardless of the displacement of the listener's head from the nominal listening position and the orientation of the listener's head relative to the nominal orientation. Therefore, when the listener's head experiences a displacement from the nominal listening position, the audio object can be perceived as moving relative to the listener's head. Similarly, when the listener's head experiences an orientation change from the nominal orientation, the audio object can be perceived as rotating relative to the listener's head. For example, the one or more speakers can be part of a headphone, or can be part of a speaker arrangement (e.g., a 2.1 speaker arrangement, a 5.1 speaker arrangement, a 7.1 speaker arrangement, etc.).

[0013] In some embodiments, modifying the object position based on the listener displacement information may be performed by translating the object position by a vector that is positively correlated with the magnitude and negatively correlated with the direction of the vector of the listener's head displacement from the nominal listening position.

[0014] Thereby, it is ensured that the nearby audio objects are perceived by the listener to move according to their head movements. This helps to provide a more realistic listening experience for these audio objects.

[0015] In some embodiments, the listener displacement information may indicate a certain small position displacement of the listener head from the nominal listening position. For example, the absolute value of the displacement may not exceed 0.5 m. The displacement may be expressed in Cartesian coordinates (e.g., x, y, z) or spherical coordinates (e.g., azimuth, elevation, radius).

[0016] In some embodiments, the listener displacement information may indicate the displacement of the listener's head from the nominal listening position, which displacement may be achieved by the listener moving his upper body and / or head. Therefore, the listener may achieve the displacement without moving his lower body. For example, when the listener sits on a chair, the displacement of the listener's head may be achieved.

[0017] In some embodiments, the position information may include an indication of the distance of the audio object from the nominal listening position. The distance (radius) may be less than 0.5 m. For example, the distance may be less than 1 cm. Alternatively, the distance of the audio object from the nominal listening position may be set to a default value by the decoder.

[0018] In some embodiments, the listener orientation information may include information about the roll, pitch, and yaw of the listener's head. The roll, pitch, and yaw may be given relative to a nominal orientation (eg, a reference orientation) of the listener's head.

[0019] In some embodiments, the listener displacement information may include information about the displacement of the listener's head from the nominal listening position expressed in Cartesian coordinates or in spherical coordinates. Thus, for Cartesian coordinates, the displacement may be expressed in x, y, z coordinates, and for spherical coordinates, the displacement may be expressed in azimuth, elevation, radius coordinates.

[0020] In some embodiments, the method may further include detecting, by a wearable device and / or a fixed device, the orientation of the listener's head. Likewise, the method may further include detecting, by a wearable device and / or a fixed device, the displacement of the listener's head from a nominal listening position. The wearable device may be, correspond to and / or include, for example, a headset or an augmented reality (AR) / virtual reality (VR) headset. For example, the fixed device may be, correspond to and / or include a camera sensor. This allows obtaining accurate information about the displacement and / or orientation of the listener's head, and thereby enabling realistic processing of nearby audio objects based on the orientation and / or displacement.

[0021] In some embodiments, the method may further include rendering the audio object to one or more real speakers or virtual speakers according to the further modified object position. For example, the audio object may be rendered to a left speaker and a right speaker of a headset.

[0022] In some embodiments, the rendering may be performed to take into account the sound occlusion of the audio object at a small distance from the listener's head based on a head-related transfer function (HRTF) of the listener's head. Thus, rendered close audio objects will be perceived by the listener in an even more realistic form.

[0023] In some embodiments, the further modified object positions may be adapted to an input format used by an MPEG-H 3D Audio renderer. In some embodiments, the rendering may be performed using an MPEG-H 3D Audio renderer. In some embodiments, the processing may be performed using an MPEG-H 3D Audio decoder. In some embodiments, the processing may be performed by a scene displacement unit of an MPEG-H 3D Audio decoder. Thus, the proposed method allows for a limited six degrees of freedom (6DoF) experience (i.e., 3DoF+) to be implemented within the framework of the MPEG-H 3D Audio standard.

[0024] According to another aspect of the present disclosure, a further method of processing position information indicating an object position of an audio object is described. The object position may be used to render the audio object. The method may include obtaining listener displacement information indicating a displacement of the listener's head. The method may further include determining the object position based on the position information. The method may further include modifying the object position based on the listener displacement information by applying a translation to the object position.

[0025] Configured as described above, the proposed method provides a more realistic listening experience, especially for audio objects positioned close to the listener's head. By being able to account for certain small translational movements of the listener's head, the proposed method enables the listener to approach close audio objects from different angles and even laterally. Thus, the proposed method can achieve an improved, more realistic immersive listening experience for the listener.

[0026] In some embodiments, modifying the object position based on the listener displacement information is performed such that, after rendering to one or more real speakers or virtual speakers according to the modified object position, the audio object is psychoacoustically perceived by the listener as originating from a position fixed relative to a nominal listening position, regardless of the displacement of the listener's head from the nominal listening position.

[0027] In some embodiments, modifying the object position based on the listener displacement information may be performed by translating the object position by a vector that is positively correlated with the magnitude and negatively correlated with the direction of the vector of the listener's head displacement from the nominal listening position.

[0028] According to another aspect of the present disclosure, a further method of processing position information indicating an object position of an audio object is described. The object position may be used to render the audio object. The method may include obtaining listener orientation information indicating an orientation of a listener's head. The method may further include determining the object position based on the position information. The method may further include modifying the object position based on the listener orientation information, such as by applying a rotational transformation to the object position (e.g., a rotation relative to the listener's head or the nominal listening position).

[0029] Configured as described above, the proposed method can take into account the orientation of the listener's head to provide the listener with a more realistic listening experience.

[0030] In some embodiments, modifying the object position based on the listener orientation information can be performed such that, after rendering to one or more real speakers or virtual speakers according to the modified object position, the audio object is psychoacoustically perceived by the listener as originating from a position fixed relative to a nominal listening position, regardless of the orientation of the listener's head relative to the nominal orientation.

[0031] According to another aspect of the present disclosure, a device for processing position information indicating an object position of an audio object is described. The object position can be used to render the audio object. The device can include a processor and a memory, the memory being coupled to the processor. The processor can be adapted to obtain listener orientation information indicating an orientation of a listener's head. The processor can be further adapted to obtain listener displacement information indicating a displacement of the listener's head. The processor can be further adapted to determine the object position based on the position information. The processor can be further adapted to modify the object position based on the listener displacement information by applying a translation to the object position. The processor can be further adapted to further modify the modified object position based on the listener orientation information, for example, by applying a rotational transformation (e.g., a rotation relative to the listener's head or the nominal listening position) to the modified object position.

[0032] In some embodiments, the processor may be adapted to modify the object position and further modify the modified object position such that, after rendering to one or more real speakers or virtual speakers according to the further modified object position, the audio object is psychoacoustically perceived by the listener as originating from a position fixed relative to a nominal listening position, regardless of the displacement of the listener's head from the nominal listening position and the orientation of the listener's head relative to the nominal orientation.

[0033] In some embodiments, the processor may be adapted to modify the object position based on the listener displacement information by translating the object position by a vector positively correlated with magnitude and negatively correlated with direction of the vector of displacement of the listener head from a nominal listening position.

[0034] In some embodiments, the listener displacement information may indicate a certain small position displacement of the listener's head from a nominal listening position.

[0035] In some embodiments, the listener displacement information may indicate the displacement of the listener's head from a nominal listening position, which displacement may be achieved by the listener moving his upper body and / or head.

[0036] In some embodiments, the position information may include an indication of the distance of the audio object from a nominal listening position.

[0037] In some embodiments, the listener orientation information may include information about the pitch, roll, and roll of the listener's head.

[0038] In some embodiments, the listener displacement information may comprise information about the displacement of the listener's head from a nominal listening position expressed in Cartesian coordinates or in spherical coordinates.

[0039] In some embodiments, the device may further include a wearable device and / or a fixed device for detecting the orientation of the listener's head. In some embodiments, the device may further include a wearable device and / or a fixed device for detecting the displacement of the listener's head from a nominal listening position.

[0040] In some embodiments, the processor may be further adapted to render the audio object to one or more real speakers or virtual speakers based on the further modified object position.

[0041] In some embodiments, the processor may be adapted to perform rendering that takes into account sound occlusion of the audio object at a small distance from the listener's head based on the HRTF of the listener's head.

[0042] In some embodiments, the processor may be adapted to adjust the further modified object position to an input format used by an MPEG-H 3D Audio renderer. In some embodiments, the rendering may be performed using an MPEG-H 3D Audio renderer. That is, the processor may implement an MPEG-H 3D Audio renderer. In some embodiments, the processor may be adapted to implement an MPEG-H 3D Audio decoder. In some embodiments, the processor may be adapted to implement a scene displacement unit of an MPEG-H 3D Audio decoder.

[0043] According to another aspect of the present disclosure, a further apparatus for processing position information indicating an object position of an audio object is described. The object position may be used to render the audio object. The apparatus may include a processor and a memory, the memory being coupled to the processor. The processor may be adapted to obtain listener displacement information indicating a displacement of the listener's head. The processor may be further adapted to determine the object position based on the position information. The processor may be further adapted to modify the object position based on the listener displacement information by applying a translation to the object position.

[0044] In some embodiments, the processor may be adapted to modify the object position based on the listener displacement information such that, after rendering to one or more real speakers or virtual speakers according to the modified object position, the audio object is psychoacoustically perceived by the listener as originating from a position fixed relative to a nominal listening position, regardless of the displacement of the listener's head from the nominal listening position.

[0045] In some embodiments, the processor may be adapted to modify the object position based on the listener displacement information by translating the object position by a vector positively correlated with magnitude and negatively correlated with direction of the vector of displacement of the listener head from a nominal listening position.

[0046] According to another aspect of the present disclosure, a further device for processing position information indicating an object position of an audio object is described. The object position can be used to render the audio object. The device can include a processor and a memory, the memory being coupled to the processor. The processor can be adapted to obtain listener orientation information indicating the orientation of a listener's head. The processor can be further adapted to determine the object position based on the position information. The processor can be further adapted to modify the object position based on the listener orientation information, for example, by applying a rotational transformation (e.g., a rotation relative to the listener's head or the nominal listening position) to the modified object position.

[0047] In some embodiments, the processor may be adapted to modify the object position based on the listener orientation information such that, after rendering to one or more real speakers or virtual speakers according to the modified object position, the audio object is psychoacoustically perceived by the listener as originating from a position fixed relative to a nominal listening position, regardless of the orientation of the listener's head relative to the nominal orientation.

[0048] According to yet another aspect, a system is described. The system may include a device according to any of the above aspects and a wearable device and / or a fixed device capable of detecting the orientation of a listener's head and detecting the displacement of the listener's head.

[0049] It should be understood that method steps and device features can be interchanged in a variety of ways. Specifically, as understood by those skilled in the art, the details of the disclosed method can be implemented as a device suitable for performing some or all of the steps of the method, and vice versa. Specifically, it should be understood that a device according to the present disclosure may relate to a device for implementing or performing a method according to the above embodiments and variants thereof, and the corresponding statements made about the method are similarly applicable to the corresponding device. Similarly, it should be understood that a method according to the present disclosure may relate to a method for operating a device according to the above embodiments and variants thereof, and the corresponding statements made about the device are similarly applicable to the corresponding method. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The present invention is explained in an exemplary manner below with reference to the accompanying drawings, in which

[0051] Figure 1 An example of an MPEG-H 3D Audio system is schematically shown;

[0052] Figure 2 An example of an MPEG-H 3D audio system according to the present invention is schematically illustrated;

[0053] Figure 3 An example of an audio rendering system according to the present invention is schematically shown;

[0054] Figure 4 A collection of examples schematically illustrating Cartesian coordinate axes and their relationship to spherical coordinates; and

[0055] Figure 5 is a flow chart schematically illustrating an example of a method for processing position information of an audio object according to the present invention. DETAILED DESCRIPTION

[0056] As used herein, 3DoF is generally a system that can correctly handle the user's head movements (particularly head rotations) specified with three parameters (e.g., yaw, pitch, roll). Such systems are typically used in various gaming systems, such as virtual reality (VR) / augmented reality (AR) / mixed reality (MR) systems, or other acoustic environments of this type.

[0057] As used herein, a user (eg, of an audio decoder or a reproduction system including an audio decoder) may also be referred to as a "listener."

[0058] As used herein, 3DoF+ shall mean that in addition to the user's head movements which can be correctly handled in a 3DoF system, certain small translational movements can also be handled.

[0059] As used herein, "certainly small" shall indicate that the movement is limited to below a threshold, typically 0.5 meters. This means that the movement from the user's original head position is no greater than 0.5 meters. For example, the user's movement is constrained by the chair he / she sits on.

[0060] As used herein, “MPEG-H 3D Audio” shall refer to the specification standardized in ISO / IEC 23008-3 and / or any future amendments, versions, or other revisions of the ISO / IEC 23008-3 standard.

[0061] In the context of the audio standards provided by the MPEG organization, the distinction between 3DoF and 3DoF+ can be defined as follows:

[0062] 3DoF: allows the user to experience side-to-side movement (e.g., the user's head), pitch movement, and roll movement.

[0063] ● 3DoF+: allows the user to experience yaw, pitch, roll and limited translational movement (of the user's head), for example, while sitting in a chair.

[0064] The limited (certainly small) head translation movement may be a movement constrained by a certain movement radius. For example, the movement may be constrained because the user is in a seated position, for example, without using the lower body. The certain small head translation movement may involve or correspond to a displacement of the user's head relative to a nominal listening position. The nominal listening position (or nominal listener position) may be a default position (e.g., a predetermined position, an expected position of the listener's head, or an optimal point for a speaker arrangement).

[0065] The 3DoF+ experience can be comparable to a limited 6DoF experience, where the panning movement can be described as limited or somewhat small head movement. In one example, the audio is also rendered with possible sound occlusion based on the user's head position and orientation. The rendering can be performed, for example, to account for sound occlusion of audio objects at small distances from the listener's head based on a head-related transfer function (HRTF) of the listener's head.

[0066] By methods, systems, devices, and other apparatuses being compatible with the functionality set forth by the MPEG-H 3D Audio standard, it may be meant that 3DoF+ is capable of being used in any future version or versions of the MPEG standards, such as a future version of the Omnidirectional Media Format (e.g., standardized in a future version of MPEG-I); and / or any update of MPEG-H Audio (e.g., an amendment or updated standard based on the MPEG-H 3D Audio standard); or any other related or companion standard that may need to be updated (e.g., a standard that specifies certain types of metadata messages and SEI messages).

[0067] For example, an audio renderer that is normative for the audio standard set forth in the MPEG-H 3D Audio Specification may be extended to include rendering of an audio scene to accurately account for user interaction with the audio scene, for example, when the user moves their head slightly sideways.

[0068] The present invention provides various technical advantages, including the advantage of providing MPEG-H 3D Audio capable of handling 3DoF+ use cases. The present invention extends the MPEG-H 3D Audio standard to support 3DoF+ functionality.

[0069] To support 3DoF+ functionality, the audio rendering system should account for a limited / small positional displacement of the user / listener's head. The positional displacement should be determined based on a relative offset from an initial position (i.e., a default position / nominal listening position). In one example, the magnitude of this offset (e.g., can be based on r offset =||P0-P1||, where P0 is the nominal listening position and P1 is the displacement position of the listener's head) The displacement position of the listener's head) is at most about 0.5m. In another example, the magnitude of the offset is limited to an offset that is only achievable when the user is sitting on a chair and does not perform lower body movements (but his head moves relative to his body). This (certainly small) offset distance results in very small (perceived) level differences and translation differences for distant audio objects. However, for close objects, even such certain small offset distances may become perceptually relevant. In fact, the listener's head movement can have a perceptual effect on the positioning of the perceived correct audio object positioning. As long as (i) the user's head displacement (e.g., r offset=||P0-P1||) to the distance to the audio object (e.g., r) trigonometrically produces an angle within the range of the user's psychoacoustic ability to detect the direction of the sound, so that this perceptual effect can remain significant (i.e., perceptually noticeable by the user / listener). Such a range may vary for different audio renderer settings, audio material, and playback configurations. For example, assuming a localization accuracy range of, e.g., + / -3°, with a left-right freedom of movement of the listener's head of + / -0.25m, this would correspond to an object distance of ~5m.

[0070] For objects close to the listener (e.g., objects at a distance <1 m from the user), correctly handling the positional displacement of the listener's head is critical for 3DoF+ scenes, as there are significant perceptual effects during both translation changes and level changes.

[0071] An example of handling of objects close to the listener is, for example, when an audio object (e.g., a mosquito) is positioned very close to the listener's face. Audio systems, such as those providing VR / AR / MR capabilities, should allow the user to perceive this audio object from all sides and angles, even if the user is making certain small translational head movements. For example, a user should be able to accurately perceive an object (e.g., a mosquito) even when the user moves their head without moving their lower body.

[0072] However, systems compliant with the current MPEG-H 3D Audio specification currently do not handle this problem correctly. Instead, using systems compliant with the MPEG-H 3D Audio system results in the "mosquito" being perceived from the wrong position relative to the user. In scenarios involving 3DoF+ performance, certain small translational movements should produce noticeable differences in the perception of audio objects (e.g., when moving a user's head to the left, a "mosquito" audio object should be perceived from the right side relative to the user's head, etc.).

[0073] The MPEG-H 3D Audio standard contains a bitstream syntax which allows object distance information to be signaled via the bitstream syntax, eg via the object_metadata() -syntax element (starting from 0.5m).

[0074] A syntax element prodMetadataConfig() may be introduced into the bitstream provided by the MPEG-H 3D Audio standard, which may be used to signal that an object is very close to the listener. For example, the syntax prodMetadataConfig() may signal that the distance between the user and the object is less than a certain threshold distance (e.g. <1 cm).

[0075] Figure 1 and Figure 2The invention is presented based on headphone rendering (ie, where the speakers move with the listener's head).

[0076] Figure 1 An example of system behavior 100 compliant with an MPEG-H 3D Audio system is shown. This example assumes that the listener's head is positioned at position P0 103 at time t0 and moves to position P1 104 at time t1>t0. The dashed circles around positions P0 and P1 indicate the allowable 3DoF+motion region (e.g., radius 0.5m). Position A 101 indicates the signaled object position (at time t0 and time t1, i.e., assuming the signaled object position is constant over time). Position A also indicates the object position rendered by the MPEG-H 3D Audio renderer at time t0. Position B 102 indicates the object position rendered by the MPEG-H 3D Audio at time t1. The vertical lines extending upward from positions P0 and P1 indicate the respective orientations (e.g., viewing directions) of the listener's head at times t0 and t1. The displacement of the user's head between position P0 and position P1 can be represented by r offset =||P0-P1|| 106. In case the listener is positioned at the default position (nominal listening position) P0 103 at time t0, he / she will perceive the audio object (e.g. mosquito) at the correct position A 101. If the user were to move to position P1 104 at time t1, he / she would perceive the audio object at position B 102 if MPEG-H 3D Audio processing is applied in its currently standardized form, which introduces the error δ shown AB 105. That is, despite the movement of the listener's head, the audio object (e.g., mosquito) will still be perceived as being located directly in front of the listener's head (i.e., moving substantially with the listener's head). It is worth noting that the introduced error δ will occur regardless of the orientation of the listener's head. AB 105.

[0077] Figure 2 An example of system behavior of the system 200 with respect to MPEG-H 3D Audio according to the present invention is shown. Figure 2, the head of the listener is positioned at position P0 203 at time t0 and moves to position P1 204 at time t1>t0. The dashed circles around positions P0 and P1 again indicate the allowable 3DoF+movement area (e.g., radius 0.5m). At 201, the indicated position A=B means the signaled object position (at time t0 and time t1, i.e., assuming the signaled object position is constant over time). Position A=B 201 also indicates the object position rendered by MPEG-H 3D Audio at time t0 and time t1. The vertical arrows extending upward from positions P0 203 and P1 204 indicate the respective orientations (e.g., viewing directions) of the listener's head at time t0 and t1. In the case where the listener is positioned at the initial / default position (nominal listening position) P0 203 at time t0, he / she will perceive the audio object (e.g., mosquito) at the correct position A 201. If the user were to move to position P1 203 at time t1, he / she would still perceive the audio object at position B 201, which is similar to (e.g., substantially equal to) position A 201 according to the present invention. Thus, the present invention allows the user's position to change over time (e.g., from position P0 203 to position P1 204) while still perceiving the sound from the same (spatially fixed) location (e.g., position A=B 201, etc.). In other words, the audio object (e.g., mosquito) moves relative to the listener's head depending on (e.g., negatively correlated with) the listener's head movements. This enables the user to move around the audio object (e.g., mosquito) and perceive the audio object from different angles or even from the side. The displacement of the user's head between position P0 and position P1 can be represented by r offset =||P0-P1||206 represents.

[0078] Figure 3 An example of an audio rendering system 300 according to the present invention is shown. The audio rendering system 300 may correspond to or include a decoder, such as an MPEG-H 3D audio decoder. The audio rendering system 300 may include an audio scene displacement unit 310 having a corresponding audio scene displacement processing interface (e.g., an interface for scene displacement data according to the MPEG-H 3D audio standard). The audio scene displacement unit 310 may output an object position 321 for rendering a corresponding audio object. For example, the scene displacement unit may output object position metadata for rendering a corresponding audio object.

[0079] The audio rendering system 300 may further include an audio object renderer 320. For example, the renderer may be composed of hardware, software, and / or any part or all of the processing performed by cloud computing, which may include various services on the Internet, generally referred to as the "cloud", that are compatible with the specifications set forth by the MPEG-H 3D audio standard, such as software development platforms, servers, storage, and software. The audio object renderer 320 may render the audio object to one or more (real or virtual) speakers according to the corresponding object positions (these object positions may be modified object positions or further modified object positions described below). The audio object renderer 320 may render the audio object to headphones and / or speakers. That is, the audio object renderer 320 may generate an object waveform according to a given reproduction format. To this end, the audio object renderer 320 may utilize compressed object metadata. Each object may be rendered to certain output channels according to its object position (e.g., modified object position, or further modified object position). Therefore, the object position may also be referred to as the channel position of its audio object. The audio object position 321 may be included in the object position metadata or scene displacement metadata output by the scene displacement unit 310.

[0080] The processing of the present invention may comply with the MPEG-H 3D Audio standard. As such, the processing may be performed by an MPEG-H 3D Audio decoder, or more specifically, by an MPEG-H scene shift unit and / or an MPEG-H 3D Audio renderer. Thus, Figure 3 The audio rendering system 300 may correspond to or include an MPEG-H 3D Audio decoder (i.e., a decoder that complies with the specifications set forth by the MPEG-H 3D Audio standard). In one example, the audio rendering system 300 may be a device including a processor and a memory coupled to the processor, wherein the processor is adapted to implement an MPEG-H 3D Audio decoder. Specifically, the processor may be adapted to implement an MPEG-H scene shift unit and / or an MPEG-H 3D Audio renderer. Thus, the processor may be adapted to perform the processing steps described in the present disclosure (e.g., as described below with reference to Figure 5 In another example, the processing or audio rendering system 300 may be performed in the cloud.

[0081] The audio rendering system 300 may obtain (eg, receive) listening localization data 301. The audio rendering system 300 may obtain the listening localization data 301 through an MPEG-H 3D audio decoder input interface.

[0082] The listening localization data 301 may indicate the orientation and / or position (eg, displacement) of the listener's head. Thus, the listening localization data 301 (which may also be referred to as posture information) may contain listener orientation information and / or listener displacement information.

[0083] The listener displacement information may indicate a displacement of a listener's head (eg, from a nominal listening position). The listener displacement information may correspond to or include an indication of a magnitude of the displacement of the listener's head from the nominal listening position, r offset =||P0-P1||206, such as Figure 2 as shown. In the context of the present invention, the listener displacement information indicates a certain small position displacement of the listener's head from the nominal listening position. For example, the absolute value of the displacement may not exceed 0.5m. Typically, this is a displacement of the listener's head from the nominal listening position, which displacement can be achieved by the listener moving his upper body and / or head. That is, the listener can achieve the displacement without moving his lower body. For example, as indicated above, the displacement of the listener's head can be achieved when the listener is sitting on a chair. The displacement can be expressed in various coordinate systems, for example, in Cartesian coordinates (expressed in x, y, z) or spherical coordinates (e.g., expressed in azimuth, elevation, radius). Alternative coordinate systems for representing the listener's head displacement are also feasible and should be understood to be covered by the present disclosure.

[0084] The listener orientation information may indicate the orientation of the listener's head (e.g., the orientation of the listener's head relative to the nominal orientation / reference orientation of the listener's head). For example, the listener orientation information may include information about the roll, pitch, and roll of the listener's head. Here, the roll, pitch, and roll may be given relative to the nominal orientation.

[0085] Listening positioning data 301 may be continuously collected from a receiver that can provide information about the user's translational movement. For example, the listening positioning data 301 used at a certain time instance may have been recently collected from the receiver. The listening positioning data 301 may be derived / collected / generated based on sensor information. For example, the listening positioning data 301 may be derived / collected / generated by a wearable device and / or a fixed device with appropriate sensors. That is, the orientation of the listener's head may be detected by the wearable device and / or the fixed device. Similarly, the displacement of the listener's head (e.g., from a nominal listening position) may be detected by the wearable device and / or the fixed device. For example, the wearable device may be, correspond to, and / or include a headset (e.g., an AR / VR headset). For example, the fixed device may be, correspond to, and / or include a camera sensor. For example, the fixed device may be included in a television or a set-top box. In some embodiments, the listening positioning data 301 may be received from an audio encoder (e.g., an encoder compliant with MPEG-H 3D audio) that may have obtained (e.g., received) sensor information.

[0086] In one example, the wearable device and / or fixed device used to detect the listening positioning data 301 can be referred to as a tracking device that supports head position estimation / detection and / or head orientation estimation / detection. There are various solutions that allow accurate tracking of the user's head movements using a computer or smartphone camera (e.g., based on facial recognition and tracking "FaceTrackNoIR", "opentrack"). Moreover, several head mounted display (HMD) virtual reality systems (e.g., HTC VIVE, Oculus Rift) have integrated head tracking technology. Any of these solutions can be used in the context of the present disclosure.

[0087] It is also important to note that the head displacement distance in the physical world does not necessarily correspond one-to-one with the displacement indicated by the listening localization data 301. To achieve hyper-realistic effects (e.g., parallax effects of overly amplified user motion), certain applications may use different sensor calibration settings or specify different mappings between motion in real space and motion in virtual space. Therefore, it can be expected that in some use cases, a certain small physical movement will produce a larger displacement in virtual reality. In any case, it can be said that the magnitude of the displacement in the physical world and virtual reality (i.e., the displacement indicated by the listening localization data 301) is positively correlated. Likewise, the direction of the displacement in the physical world and virtual reality is positively correlated.

[0088] The audio rendering system 300 may further receive (object) position information (e.g., object position data) 302 and audio data 322. The audio data 322 may include one or more audio objects. The position information 302 may be part of the metadata of the audio data 322. The position information 302 may indicate the corresponding object positions of the one or more audio objects. For example, the position information 302 may include an indication of the distance of the corresponding audio object relative to the nominal listening position of the user / listener. The distance (radius) may be less than 0.5m. For example, the distance may be less than 1cm. If the position information 302 does not include an indication of the distance of a given audio object from the nominal listening position, the audio rendering system may set the distance of this audio object from the nominal listening position to a default value (e.g., 1m). The position information 302 may further include an indication of the elevation and / or azimuth of the corresponding audio object.

[0089] Each object position can be used to render its corresponding audio object. Therefore, the position information 302 and the audio data 322 can be included in or form object-based audio content. The audio content (e.g., the audio object / audio data 322 and its position information 302) can be transmitted in an encoded audio bitstream. For example, the audio content can be in the format of a bitstream received from a network transmission. In this case, it can be said that the audio rendering system receives the audio content (e.g., from an encoded audio bitstream).

[0090] In one example of the present invention, metadata parameters may be used to correct the handling of use cases with backwards compatible enhancements for 3DoF and 3DoF+. In addition to listener orientation information, metadata may also contain listener displacement information. Such metadata parameters may be provided by Figure 2 and 3 And any other embodiment of the system shown in the present invention is utilized.

[0091] Backwards compatibility enhancements may allow correct handling of use cases (e.g., embodiments of the present invention) based on the normative MPEG-H 3D Audio scene displacement interface. This means that a conventional MPEG-H 3D Audio decoder / renderer will still produce output, albeit incorrect output. However, an enhanced MPEG-H 3D Audio decoder / renderer according to the present invention will correctly apply the extension data (e.g., extended metadata) and processing, and may therefore handle scenes with objects positioned close to the listener in a correct manner.

[0092] In one example, the invention involves providing data for certain small translational movements of the user's head in a format different from that outlined below, and possibly adapting the formulas accordingly. For example, rather than in the format of azimuth, elevation and radius (in a spherical coordinate system), the data may be provided in a format such as x-coordinate, y-coordinate, z-coordinate (in a Cartesian coordinate system). Examples of these coordinate systems relative to each other are as follows: Figure 4 shown.

[0093] In one example, the present invention relates to providing metadata (e.g., contained in Figure 3 The metadata may be used, for example, to interface with scene displacement data. The metadata (eg, listener displacement information) may be obtained by deploying a tracking device that supports 3DoF+ or 6DoF tracking.

[0094] In one instance, metadata (e.g., listener displacement information, specifically the displacement of the listener's head, or equivalently, scene displacement) can be represented by the following three parameters: sd_azimuth, sd_elevation, and sd_radius, which relate to the azimuth, elevation, and radius (spherical coordinates) of the displacement of the listener's head (or scene displacement).

[0095] The syntax for these parameters is given in the following table.

[0096] Table 264b—Mpegh3daPositionalSceneDisplacementData() Syntax

[0097]

[0098] sd_azimuth This field defines the scene displacement azimuth position. This field can take values ​​from -180 to 180.

[0099] az offset =(sd_azimuth-128)·1.5

[0100] az offset =min(max(az offset ,-180),180)

[0101] sd_elevation This field defines the scene displacement elevation position. This field can take values ​​from -90 to 90.

[0102] el offset =(sd_elevation-32)·3.0

[0103] el offset=min(max(el offset ,-90),90)

[0104] sd_radius This field defines the scene displacement radius. This field can take values ​​from 0.015626 to 0.25.

[0105] r offset =(sd_radius+1) / 16

[0106] In another example, metadata (e.g., listener displacement information) may be represented by the following three parameters in Cartesian coordinates: sd_x, sd_y, and sd_z, which reduces the processing of the data from spherical coordinates to Cartesian coordinates. The metadata may be based on the following syntax:

[0107]

[0108] As described above, the above syntax or its equivalent syntax can be used to signal information related to rotation around the x-axis, y-axis, and z-axis.

[0109] In one example of the invention, processing of scene displacement angles of channels and objects can be enhanced by extending equations that account for changes in the position of the user's head. That is, processing of object positions can take into account (eg, can be based at least in part on) listener displacement information.

[0110] Figure 5 An example of a method 500 for processing position information indicating an object position of an audio object is shown in a flowchart of FIG. This method may be performed by a decoder, such as an MPEG-H 3D audio decoder. Figure 3 The audio rendering system 300 can serve as an example of such a decoder.

[0111] As a first step ( Figure 5 ), for example, receiving audio content including audio objects and corresponding position information from a bit stream of encoded audio. Then, the method may further include decoding the encoded audio content to obtain the audio objects and the position information.

[0112] exist Step S510 At the location, listener orientation information is obtained (eg, received). The listener orientation information may indicate the orientation of the listener's head.

[0113] exist Step S520 At the address, listener displacement information is obtained (eg, received). The listener displacement information may indicate the displacement of the listener's head.

[0114] exist Step S530At the location of the audio object, the object position is determined based on the position information. For example, the object position can be extracted from the position information (e.g., expressed as an azimuth, elevation, radius, or x, y, z, or their equivalents). The determination of the object position can also be based at least in part on information about the geometry of the speaker arrangement of one or more (real or virtual) speakers in the listening environment. If the radius is not included in the position information of the audio object, the decoder can set the radius to a default value (e.g., 1 m). In some embodiments, the default value may depend on the geometry of the speaker arrangement.

[0115] It is worth noting that steps S510 , S520 , and S521 may be performed in any order.

[0116] exist Step S540 At step S530, the object position determined at step S530 is modified based on the listener displacement information. This may be done by applying a translation to the object position based on the displacement information (e.g., based on the displacement of the listener's head). Thus, it may be said that modifying the object position involves correcting the object position for the displacement of the listener's head (e.g., the displacement from the nominal listening position). Specifically, modifying the object position based on the listener displacement information may be performed by translating the object position by a vector that is positively correlated with the magnitude and negatively correlated with the direction of the vector of the displacement of the listener's head from the nominal listening position. Figure 2 An example of such a translation is schematically shown in .

[0117] exist Step S550 At step S540, the modified object position obtained at step S540 is further modified based on the listener orientation information. For example, this can be done by applying a rotational transformation to the modified object position according to the listener orientation information. This rotation can be, for example, a rotation relative to the listener's head or nominal listening position. The rotational transformation can be performed by a scene displacement algorithm.

[0118] As noted above, when applying a rotational transform, user offset compensation (i.e., modification of object positions based on listener displacement information) is taken into account. For example, applying a rotational transform may include:

[0119] ● Calculate the rotation transformation matrix (based on the user orientation, e.g., listener orientation information),

[0120] ●Convert the object position from spherical coordinates to Cartesian coordinates;

[0121] ● applying a rotational transformation to the user-position-offset compensated audio object (i.e., to the modified object position), and

[0122] ●After the rotation transformation, convert the object position from Cartesian coordinates back to spherical coordinates.

[0123] As another Step S560 ( Figure 5 (not shown in the figure), the method 500 may include rendering the audio object to one or more real speakers or virtual speakers according to the further modified object position. To this end, the further modified object position may be adjusted to an input format used by an MPEG-H 3D audio renderer (e.g., the audio object renderer 320 described above). The one or more (real or virtual) speakers may be, for example, part of a headset, or may be part of a speaker arrangement (e.g., a 2.1 speaker arrangement, a 5.1 speaker arrangement, a 7.1 speaker arrangement, etc.). In some embodiments, for example, the audio object may be rendered to a left speaker and a right speaker of a headset.

[0124] The purpose of steps S540 and S550 described above is as follows. That is, modifying the object position and further modifying the modified object position is performed so that after rendering to one or more (real or virtual) speakers according to the further modified object position, the audio object is psychoacoustically perceived by the listener as originating from a fixed position relative to the nominal listening position. This fixed position of the audio object should be psychoacoustically perceived regardless of the displacement of the listener's head from the nominal listening position and regardless of the orientation of the listener's head relative to the nominal orientation. In other words, when the listener's head experiences a displacement from the nominal listening position, the audio object can be perceived as moving (translating) relative to the listener's head. Similarly, when the listener's head experiences a change in orientation from the nominal orientation, the audio object can be perceived as moving (rotating) relative to the listener's head. Thus, the listener can perceive a nearby audio object from different angles and distances by moving his head.

[0125] Modifying the object position at steps S540 and S550 and further modifying the modified object position, respectively, may be performed in the context of (rotation / translation) of an audio scene shift, for example, by the audio scene shift unit 310 described above.

[0126] It should be noted that certain steps may be omitted depending on the particular use case at hand. For example, if the listening localization data 301 only contains listener displacement information (but no listener orientation information, or only contains listener orientation information indicating that the orientation of the listener's head has not deviated from the nominal orientation), step S550 may be omitted. The rendering at step S560 will then be performed according to the modified object positions determined at step S540. Similarly, if the listening localization data 301 only contains listener orientation information (but no listener displacement information, or only contains listener displacement information indicating that the position of the listener's head has not deviated from the nominal listening position), step S540 may be omitted. Step S550 will then involve modifying the object positions determined at step S530 based on the listener orientation information. The rendering at step S560 will be performed according to the modified object positions determined at step S550.

[0127] Broadly speaking, the present invention provides for position updating of object positions received as part of object-based audio content (eg, position information 302 and audio data 322 ) based on a listener's listening localization data 301 .

[0128] First, the object position (or channel position) p=(az,el,r) is determined. This may be performed in the context of step 530 of method 500 (eg, as part of said step).

[0129] For a channel-based signal, the radius r can be determined as follows:

[0130] - If the intended loudspeaker (of a channel based on the input signal of the channel) exists in a reproduction loudspeaker setup, and the distance of the reproduction setup is known, set the radius r to the loudspeaker distance (eg in cm).

[0131] - If the intended loudspeaker is not present in the reproduction loudspeaker setup, but the distance of the reproduction loudspeaker (eg from the nominal listening position) is known, then the radius r is set to the maximum reproduction loudspeaker distance.

[0132] - If the expected speaker is not present in the reproduction speaker setup and the reproduction speaker distance is not known, the radius r is set to a default value (eg, 1023 cm).

[0133] For object-based signals, the radius r is determined as follows:

[0134] - If the object distance is known (e.g. known from the production tool and production format and communicated in prodMetadataConfig()), then the radius r is set to the known object distance (e.g. signaled by goa_bsObjectDistance[] (in cm) according to table AMD5.7 of the MPEG-H 3D Audio standard).

[0135] Table AMD5.7—goa_Production_Metadata() syntax

[0136]

[0137] - If the object distance is known from the position information (e.g., known from the object metadata and transmitted in object_metadata()), then the radius r is set to the object distance signaled in the position information (e.g., to radius[] (in cm) transmitted with the object metadata). The radius r can be signaled as described in the sections "Scaling of object metadata" and "Limiting object metadata" below.

[0138] Scaling of object metadata

[0139] As an optional step in the context of determining the position of an object, the object position p=(az,el,r) determined from the position information may be scaled. This may involve applying a scaling factor to invert the encoder scaling of the input data for each component. This may be performed for each object. The actual scaling of the object position may be implemented according to the following pseudo code:

[0140]

[0141]

[0142] Restricting Object Metadata

[0143] As a further optional step in the context of determining the position of an object, the (possibly scaled) object position p=(az,el,r) determined from the position information may be constrained. This may involve applying constraints on the decoded values ​​for each component so that the values ​​remain within a valid range. This may be performed for each object. The actual constraints on the object position may be implemented according to the functionality of the following pseudo code:

[0144]

[0145]

[0146] The determined (and optionally scaled and / or constrained) object position p=(az,el,r) may then be converted to a predetermined coordinate system, e.g., a coordinate system according to a "common convention" where 0° azimuth is at the right ear (positive values ​​are counterclockwise) and 0° elevation is at the top of the head (positive values ​​are downward). Thus, the object position p may be converted to a position p' according to the "common" convention. This yields the object position p' using:

[0147] p′=(az',el',r)

[0148] az′=az+90°

[0149] el′=90°-el

[0150] The radius r remains unchanged.

[0151] At the same time, the listener's head displacement information (az offset ,e loffset ,r offset ) into a predetermined coordinate system. Using the "common convention", this is equivalent to

[0152] az′ offset =az offset +90°

[0153] el′ offset =90°-el offset

[0154] The radius r offset constant.

[0155] It is worth noting that the conversion to a predetermined coordinate system for both the object position and the listener head displacement may be performed in the context of step S530 or step S540 .

[0156] The actual location update may be performed in the context of (eg, as part of) step S540 of method 500. The location update may include the following steps:

[0157] As a first step, the position p, or the position p′ in case the transfer to the predetermined coordinate system has been performed, is transferred to Cartesian coordinates (x, y, z). In the following, without intended limitations, the process will be described for the position p′ in the predetermined coordinate system. Moreover, without intended limitations, the following orientation / direction of the coordinate axes may be assumed: the x-axis points to the right (when in the nominal orientation, viewed from the listener's head), the y-axis points straight ahead, and the z-axis points straight up. At the same time, the listener displacement information (az′) of the listener's head may be transferred to the listener's head. offset ,el′ offset ,roffste ) is converted into Cartesian coordinates.

[0158] As a second step, the object positions in Cartesian coordinates are offset (translated) according to the displacement of the listener's head (scene displacement) in the manner described above. This can be done as follows:

[0159] x=r·sin(el′)·cos(az′)+r offset ·sin(el′ offset )·cos(az′ offset )

[0160] y=r·sin(el′)·sin(az′)+r offset ·sin(el′ offset )·sin(az′ offset )

[0161] z=r·cos(el′)+r offset ·cos(el′ offset )

[0162] The above translation is an example of modifying the position of the object based on the listener displacement information in step S540 of method 500 .

[0163] The offset object position in Cartesian coordinates is converted to spherical coordinates and may be referred to as p". The offset object position may be expressed in a predetermined coordinate system according to a common convention as p" = (az", el", r').

[0164] When there is a listener head displacement that produces a certain small radius parameter change (ie, r′≈r), the modified object position p″ can be redefined as p″=(az″,el″,r).

[0165] In another example, when there is a large listener head displacement that may produce a considerable radius parameter variation (i.e., r′>>r), the modified object position p″ may also be defined as p″=(az″,el″,r′) instead of p″=(az″,el″,r) with a modified radius parameter r′.

[0166] The distance from the listener's head displacement (i.e., r offset =||P0-P1||) and the initial radius parameter (ie, r=||P0-A||) to obtain the corresponding value of the modified radius parameter r' (see, for example, Figure 1 and 2 ). For example, the modified radius parameter r′ can be determined based on the following trigonometric relationship:

[0167]

[0168] Mapping this modified radius parameter r′ to object / channel gains and its application in subsequent audio rendering can significantly improve the perceived effect of level changes due to user movement. Such modifications to the radius parameter r′ are allowed to achieve an "adaptive sweet spot". This would mean that the MPEG rendering system dynamically adjusts the sweet spot position based on the current positioning of the listener. In general, the rendering of audio objects according to the modified (or further modified) object position can be based on the modified radius parameter r′. In particular, the object / channel gain for rendering the audio object can be based on the modified radius parameter r′ (e.g., modified based on the modified radius parameter).

[0169] In another example, during speaker reproduction setup and rendering (e.g., The above step S560 ), scene displacement may be disabled. However, optional enabling of scene displacement may be available. This enables the 3DoF+ renderer to create a dynamically adjustable sweet spot based on the listener's current position and orientation.

[0170] It is worth noting that the step of converting the object positions and the displacement of the listener's head into Cartesian coordinates is optional, and the translation / offset (modification) according to the displacement of the listener's head (scene displacement) can be performed in any suitable coordinate system. In other words, the choice of Cartesian coordinates in the above text should be understood as a non-limiting example.

[0171] In some embodiments, scene displacement processing (including modifying object positions and / or further modifying modified object positions) can be enabled or disabled by a flag (field, element, setting bit) in the bitstream (e.g., a useTrackingMode element). Subclauses "17.3 Interface for local speaker setup and rendering" and "17.4 Interface for binaural room impulse response (BRIR)" in ISO / IEC 23008-3 contain a description of the element useTrackingMode that activates scene displacement processing. In the context of the present disclosure, the useTrackingMode element shall define (subclause 17.3) whether processing of scene displacement values ​​sent through the mpegh3daSceneDisplacementData() interface and the mpegh3daPositionalSceneDisplacementData() interface occurs. Alternatively or additionally, (subclause 17.4) the useTrackingMode field shall define whether a tracker device is connected and whether binaural rendering shall be handled in a special head tracking mode, which means that processing of scene displacement values ​​sent through the mpegh3daSceneDisplacementData() interface and the mpegh3daPositionalSceneDisplacementData() interface shall take place.

[0172] The method and system described herein can be implemented as software, firmware and / or hardware. Certain components can, for example, be implemented as software running on a digital signal processor or a microprocessor. Other components can, for example, be implemented as hardware and / or an application specific integrated circuit. The signals encountered in the described method and system can be stored on media such as random access memory or optical storage media. The signal can be transmitted through a network such as a radio network, a satellite network, a wireless network or a wired network, for example, the Internet. Typical devices utilizing the method and system described herein are portable electronic devices or other consumer devices for storing and / or rendering audio signals.

[0173] Although this document makes reference to MPEG, and in particular MPEG-H 3D Audio, the present disclosure should not be construed as being limited to these standards. On the contrary, as will be appreciated by those skilled in the art, the present disclosure may also find advantageous application in other audio coding standards.

[0174] Furthermore, while this document frequently refers to certain small positional displacements of a listener's head (eg, from a nominal listening position), the present disclosure is not limited to certain small positional displacements and may generally be applied to any positional displacement of a listener's head.

[0175] It should be noted that the description and the accompanying drawings only illustrate the principles of the proposed method, system and device. Those skilled in the art will be able to implement various arrangements, which, although not explicitly described or shown herein, embody the principles of the present invention and are included in the spirit and scope of the present invention. In addition, all examples and embodiments outlined in this document are clearly intended in principle to be for explanation purposes only, to help the reader understand the principles of the proposed method. In addition, all statements of the principles, aspects and embodiments of the present invention and their specific examples provided herein are intended to cover their equivalents.

[0176] In addition to the above, various example embodiments and example examples of the present invention will become apparent from the following listing of enumerated example examples (EEE), which are not claims.

[0177] The first EEE relates to a method for decoding an encoded audio signal bitstream, the method comprising: receiving, by an audio decoding device 300, the encoded audio signal bitstream 302, 322, wherein the encoded audio signal bitstream comprises encoded audio data 322 and metadata corresponding to at least one object-audio signal 302; decoding, by the audio decoding device 300, the encoded audio signal bitstream 302, 322 to obtain a representation of a plurality of sound sources; receiving, by the audio decoding device 300, listening positioning data 301; generating, by the audio decoding device 300, audio object position data 321, wherein the audio object position data 321 describes a plurality of sound sources relative to a listening positioning based on the listening positioning data 301.

[0178] A second EEE relates to the method of the first EEE, wherein the listening localization data 301 is based on a first set of first pan displacement data and a second set of second pan displacement and orientation data.

[0179] A third EEE relates to the method of the second EEE, wherein the first translation displacement data or the second translation displacement data is based on at least one of a spherical coordinate set or a Cartesian coordinate set.

[0180] A fourth EEE relates to the method of the first EEE, wherein the listening localization data 301 is obtained via an MPEG-H 3D audio decoder input interface.

[0181] A fifth EEE relates to the method of the first EEE, wherein the encoded audio signal bitstream comprises an MPEG-H 3D audio bitstream syntax element, and wherein the MPEG-H 3D audio bitstream syntax element comprises the encoded audio data 322 and the metadata corresponding to at least one object-audio signal 302.

[0182] A sixth EEE relates to the method of the first EEE, the method further comprising rendering, by the audio decoding device 300 , the plurality of sound sources to a plurality of speakers, wherein the rendering process at least complies with the MPEG-H 3D audio standard.

[0183] A seventh EEE relates to the method of the first EEE, the method further comprising converting, by the audio decoding device 300 , a position p corresponding to the at least one object-audio signal 302 into a second position p″ corresponding to the audio object position 321 based on a translation of the listening localization data 301 .

[0184] An eighth EEE relates to the method of the seventh EEE, wherein a position p' of the audio object position in a predetermined coordinate system is determined based on the following (eg, according to a common convention):

[0185] P'=(az',el',r)

[0186] az′=az+90°

[0187] el′=90°-el

[0188] az′ offset =az offset +90°

[0189] el′ offset =90°-el offset

[0190] Where ax corresponds to a first azimuth parameter, el corresponds to a first elevation parameter, and r corresponds to a first radius parameter, herein ax′ corresponds to a second azimuth parameter, el′ corresponds to a second elevation parameter and r′ corresponds to a second radius parameter, where az offset Corresponding to the third-party angular parameter, el offset corresponds to the third elevation parameter, and where az′ offset Corresponding to the fourth azimuth parameter, el′ offset Corresponding to the fourth elevation angle parameter.

[0191] A ninth EEE relates to the method of the eighth EEE, wherein the offset audio object position p″ 321 of the audio object position 302 is determined in Cartesian coordinates (x, y, z) based on:

[0192] x=r·sin(el′)·cos(az′)+x offset

[0193] y=r·sin(el′)·sin(az′)+y offset

[0194] z=r·cos(el′)+z offset

[0195] Wherein the Cartesian position (x, y, z) is composed of an x ​​parameter, a y parameter and a z parameter, and where x offset Involves the first x-axis offset parameter, y offset involves the first y-axis offset parameter, and z offset Related to the first z-axis offset parameter.

[0196] The tenth EEE relates to the method of the ninth EEE, wherein the parameter x offset ,y offset , and z offset Based on the following:

[0197] x offset =r offset ·sin(el′ offset )·cos(az′ offset )

[0198] y offset =r offset ·sin(el′ offset )·sin(az′ offset )

[0199] z offset =r offset ·cos(el′ offset )

[0200] An eleventh EEE relates to the method of the seventh EEE, wherein the azimuth parameter az offset It involves the scene displacement azimuth position and is based on:

[0201] az offset =(sd_azimuth-128)·1.5

[0202] az offset =min(max(az offset ,-180),180)

[0203] where sd_azimuth is the azimuth metadata parameter indicating the MPEG-H 3DA azimuth scene displacement, where the elevation parameter el offset It involves the scene displacement elevation position and is based on the following:

[0204] el offset =(sd_elevation-32)·3

[0205] el offset=min(max(el offset ,-90),90)

[0206] where sd_elevation is the elevation metadata parameter indicating the MPEG-H 3DA elevation scene displacement, where the radius parameter r offset Refers to the scene displacement radius and is based on:

[0207] r offset =(sd_radius+1) / 16

[0208] where sd_radius is a radius metadata parameter indicating the MPEG-H 3DA radial scene displacement, and where parameters X and Y are scalar variables.

[0209] The twelfth EEE relates to the method of the tenth EEE, wherein the x offset The parameter relates to the scene displacement offset sd_x in the x-axis direction; the y offset The parameter relates to the scene displacement offset displacement sd_y in the y-axis direction; and the z offset The parameter refers to the scene displacement offset sd_z in the z-axis direction.

[0210] A thirteenth EEE relates to the method of the first EEE, the method further comprising interpolating, by the audio decoding device, the first position data associated with the listening localization data 301 and the object audio signal 302 at an update rate.

[0211] A fourteenth EEE relates to the method of the first EEE, the method further comprising determining, by the audio decoding device 300 , an efficient entropy coding of listening localization data 301 .

[0212] A fifteenth EEE relates to the method of the first EEE, wherein the position data associated with listening localization data 301 are derived based on sensor information.

Claims

1. A method of processing position information indicating an object position of an audio object, wherein the processing is performed using an MPEG-H 3D Audio decoder, wherein the object position can be used for rendering the audio object, the method comprising: Obtaining listener orientation information indicating an orientation of a listener's head, wherein the listener orientation information includes information about a roll, pitch, and yaw of the listener's head; obtaining, via an MPEG-H 3D audio decoder input interface, listener displacement information indicating a displacement of the listener's head relative to a nominal listening position; determining the position of the object according to the position information; modifying the object position based on the listener displacement information by applying a translation to the object position; as well as The modified object position is further modified based on the listener orientation information, wherein when the listener displacement information indicates that the listener head is displaced by a certain small position displacement from the nominal listening position, the absolute value of the certain small position displacement is 0.5 meters or less than 0.5 meters, and the distance between the audio object position after the listener head displacement and the listening position is equal to the distance between the modified object position and the nominal listening position.

2. A non-transitory computer-readable medium comprising instructions, which, when executed by a digital signal processor or a microprocessor, cause the digital signal processor or the microprocessor to perform the method of claim 1.

3. An MPEG-H 3D audio decoder for processing position information indicating an object position of an audio object, wherein the object position can be used to render the audio object, the decoder comprising a processor and a memory, the memory being coupled to the processor, wherein the processor is adapted to: Obtaining listener orientation information indicating an orientation of a listener's head, wherein the listener orientation information includes information about a roll, pitch, and yaw of the listener's head; obtaining, via an MPEG-H 3D audio decoder input interface, listener displacement information indicating a displacement of the listener's head relative to a nominal listening position; determining the position of the object according to the position information; modifying the object position based on the listener displacement information by applying a translation to the object position; as well as The modified object position is further modified based on the listener orientation information, wherein When the listener displacement information indicates that the listener head is displaced by a certain small position displacement from the nominal listening position, the absolute value of the certain small position displacement is 0.5 meters or less than 0.5 meters, wherein the distance between the audio object position after the listener head displacement and the listening position is equal to the distance between the modified object position and the nominal listening position.

4. A method for decoding an encoded audio signal bitstream, the method comprising: receiving the encoded audio signal bitstream, wherein the encoded audio signal bitstream comprises encoded audio data and metadata corresponding to at least one object-audio signal; decoding the encoded audio signal bitstream to obtain representations of a plurality of sound sources; receiving listening location data; and Audio object position data is generated, wherein the audio object position data describes a plurality of sound sources relative to a listening position based on the listening position data.

Images