Modification of spatial audio scenes

Abstract

An apparatus comprising means for identifying a fan-shaped portion of an audio scene wherein the fan-shaped portion comprises one or more persons; and determining a modification to the spatial audio content for rendering the audio scene to reduce an angular offset between a scene orientation of the audio scene for rendering and a sector orientation of the identified sector.

Description

Technical Field

[0001] Examples of the present disclosure are directed to enabling controlled modification of spatial audio scenes. Background Art

[0002] Spatial audio refers to audio where the audio sources are distinctively localized, usually in three dimensions. Localization can be directional only, but can also be positional (direction and distance).

[0003] The captured audio scene captures audio sources at different locations in the audio space. It is captured as spatial audio content using multiple microphones (such as a microphone array).

[0004] The rendered audio scene renders audio sources at different locations in the rendered audio space. It is rendered from spatial audio content using multiple speakers (e.g., using a speaker setup or via headphones). The headphones may have head tracking capabilities. The audio (audio scene) rendered to the listener depends on the listener's position and / or orientation in the rendered audio space, and in some examples, this may track the listener's position and / or orientation in real space.

[0005] It is desirable to improve the perception of an audio scene as it is rendered to a listener. Summary of the invention

[0006] According to various but not necessarily all examples, there is provided an apparatus comprising means for performing the following operations:

[0007] identifying a sector-shaped portion of an audio scene, wherein the sector-shaped portion includes one or more persons and the sector-shaped portion has an angular extension that does not extend beyond a spatial distribution of the persons in the audio scene; and

[0008] A modification to the spatial audio content is determined for use in rendering the audio scene to reduce an angular offset between a scene orientation of the rendered audio scene and a sector orientation of the identified sector.

[0009] In some but not necessarily all examples, a scene orientation for a rendered audio scene is defined by a front direction for rendering the audio scene, and a sector orientation of the identified sector is defined by a midline of the sector.

[0010] In some but not necessarily all examples, the component for identifying a sector of an audio scene that includes one or more people is configured to identify the sector as an active sector and includes a component for processing audio content captured from the audio scene to identify a sector of the audio scene that actively includes a speech audio source.

[0011] In some but not necessarily all examples, the component for identifying a sector-shaped portion of an audio scene that includes one or more persons is configured to identify the sector-shaped portion as an active sector-shaped portion and includes a component for receiving input from one or more person-presence-sensors for the audio scene, which detect the active physical presence of one or more persons in the sector-shaped portion of the audio scene.

[0012] In some but not necessarily all examples, the component for identifying a sector-shaped portion of an audio scene that includes one or more persons is configured to identify the sector-shaped portion as an active sector-shaped portion and includes one or more in-vehicle person-presence-sensors, which are configured to detect the active physical presence of one or more persons in the sector-shaped portion of the audio scene, wherein the audio scene is an in-vehicle audio scene.

[0013] In some but not necessarily all examples, the one or more vehicle occupant presence sensors include a vehicle seat weight sensor and / or a vehicle seat belt buckle activation sensor.

[0014] In some but not necessarily all examples, means for determining a modification to spatial audio content for rendering an audio scene to reduce an angular offset between a scene orientation of the audio scene and a sector orientation of an identified sector includes means for rotating the audio scene.

[0015] In some but not necessarily all examples, the apparatus includes a modification component for modifying spatial audio content for rendering an audio scene, wherein the spatial audio content includes at least a first audio source at a first orientation relative to a scene orientation of the audio scene and a second audio source at a second orientation relative to the scene orientation of the audio scene, wherein the modification component is configured to reduce an angular offset between the first orientation and the scene orientation of the audio scene, and an angular offset between the second orientation and the scene orientation of the audio scene.

[0016] In some but not necessarily all examples, the modifying component is configured to modify the spatial audio content such that a scene orientation of the audio scene is intermediate the first orientation and the second orientation.

[0017] In some but not necessarily all examples, a component for determining a modification to spatial audio content for rendering an audio scene to reduce an angular offset between a scene orientation of the audio scene and an orientation of a sector of an identified sector includes a modification component for modifying the spatial audio content for rendering the audio scene to optimize a cost function for an interval distribution of orientations favorable to a person with respect to the scene orientation of the audio scene.

[0018] In some but not necessarily all examples, means for determining a modification to the spatial audio content for rendering an audio scene to reduce an angular offset between a scene orientation of the audio scene and a sector orientation of the identified sector includes:

[0019] means for determining a modification to the spatial audio content for rendering the audio scene to reduce to zero an angular offset between a scene orientation of the audio scene and a sector orientation of the identified sector; or

[0020] Means for determining a modification to the spatial audio content for rendering the audio scene to reduce an angular offset between a scene orientation of the audio scene and a sector orientation of the identified sector to a non-zero value.

[0021] In some but not necessarily all examples, a microphone for capturing spatial audio from an audio scene is fixed relative to the audio scene, and wherein components for determining modifications to the spatial audio content for rendering the audio scene to reduce an angular offset between a scene orientation of the audio scene and a sector orientation of an identified sector include: components for virtually reorienting the microphone.

[0022] In some but not necessarily all examples, the apparatus is configured to transmit a modification record of the modification of the spatial audio content to reduce a difference between the orientation of the audio source and the scene orientation of the audio scene to enable or not to make the modification.

[0023] According to various but not necessarily all examples, a method is provided, comprising:

[0024] identifying a sector-shaped portion of an audio scene, wherein the sector-shaped portion includes one or more persons and the sector-shaped portion has an angular extension that does not extend beyond a spatial distribution of the persons in the audio scene; and

[0025] A modification to the spatial audio content is determined for use in rendering the audio scene to reduce an angular offset between a scene orientation of the rendered audio scene and a sector orientation of the identified sector.

[0026] According to various but not necessarily all examples, there is provided a computer program comprising instructions which, when executed by one or more processors, cause:

[0027] identifying a sector-shaped portion of an audio scene, wherein the sector-shaped portion includes one or more persons and the sector-shaped portion has an angular extension that does not extend beyond a spatial distribution of the persons in the audio scene; and

[0028] A modification to the spatial audio content is determined for use in rendering the audio scene to reduce an angular offset between a scene orientation of the rendered audio scene and a sector orientation of the identified sector.

[0029] According to various but not necessarily all examples, there is provided an apparatus comprising means for performing the following operations:

[0030] identifying a sector-shaped portion of an audio scene, wherein the sector-shaped portion includes one or more persons and the sector-shaped portion has an angular extension that does not extend beyond a spatial distribution of the persons in the audio scene;

[0031] determining a modification to the spatial audio content for rendering the audio scene to reduce an angular offset between a scene orientation of the rendered audio scene and a sector orientation of the identified sector; and

[0032] Based on the determined modification, the spatial audio content is modified for rendering the audio scene to reduce, in the rendered audio scene, an angular offset between a scene direction of the rendered audio scene and a sector orientation of the identified sector.

[0033] Examples are provided as claimed in the following claims according to various but not necessarily all examples.

[0034] Although the above examples and optional features of the present disclosure are described separately, it should be understood that they are included in the present disclosure in all possible combinations and arrangements. It should be understood that the various examples of the present disclosure may include any or all of the features described for other examples of the present disclosure, and vice versa. In addition, it should be understood that any one or more or all of the features (in any combination) can be implemented by devices, methods, and / or computer program instructions as needed and as appropriate, or included in devices, methods, and / or computer program instructions, or can be performed by devices, methods, and / or computer program instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Some examples will now be described with reference to the accompanying drawings, in which:

[0036] Figure 1A An example of an audio scene 20 before modification is shown, Figure 1BThe modified audio scene 20 is shown;

[0037] Figure 2A An example of the audio scene 20 before modification is shown, Figure 2B shows the audio scene 20 after modification using the cost function;

[0038] Figure 3 An example of an apparatus 10 for determining a modification to spatial audio content to modify a rendered spatial audio scene is shown;

[0039] Figure 4 A method 100 of adaptive spatial audio capture comprising determining a modification to spatial audio content to modify a rendered spatial audio scene is shown;

[0040] Figure 5 An example of a system suitable for conducting a telecommunication session, such as an immersive voice call, is shown;

[0041] Fig. 6A and Figure 6B An example of in-vehicle spatial audio capture is shown using two different views of the same vehicle. Fig. 7A and Figure 7B An example of in-vehicle spatial audio rendering is shown using two different views of the same vehicle;

[0042] Figure 8 and Fig. 9 shows the use of sector sections for in-vehicle audio capture;

[0043] Fig. 10A shows the original in-vehicle sound scene as captured from the front microphone array;

[0044] Fig. 10B The original in-vehicle sound scene is shown rendered without modification;

[0045] Fig.11A showing modifications to the original in-vehicle sound scene;

[0046] Fig. 11B showing a rendered in-vehicle sound scene with modifications;

[0047] Fig. 12A shows the original in-vehicle sound scene as captured from the central microphone array;

[0048] Fig. 12B The original in-vehicle sound scene is shown rendered without modification;

[0049] Fig.13A showing modifications to the original in-vehicle sound scene;

[0050] Fig. 13B showing a rendered in-vehicle sound scene with modifications;

[0051] Fig.14A showing different modifications to the original in-vehicle sound scene;

[0052] Fig. 14B showing rendered in-vehicle sound scenes with different modifications;

[0053] Fig.15A , Fig. 15B and Fig. 15C Shows the use of head tracking ( Fig.15A , 15B ) and without head tracking ( Fig.15A , 15C )'s headphones 52 to render an example of an audio scene;

[0054] Fig.16 Shows an example of using the IVAS decoder / renderer;

[0055] Fig.17 An example of a controller 400 suitable for determining and / or applying modifications is shown;

[0056] Fig.18 An example of a computer program suitable for determining and / or applying the modification is shown;

[0057] The drawings are not necessarily drawn to scale. For the sake of clarity and brevity, certain features and views in the drawings may be shown schematically or exaggerated in scale. For example, the dimensions of certain elements in the drawings may be exaggerated relative to other elements to aid in illustration. Similar reference numbers are used in the drawings to indicate similar features. For the sake of clarity, all reference numbers may not be shown in all drawings.

[0058] In the following description, reference numbers without subscript indexes (e.g., 10) may be used to refer to a class (or group / collection), reference numbers with numeric subscript indexes (e.g., 10_1) may be used to refer to specific instances of the class (members of the group / collection), and reference numbers with variable subscript indexes (e.g., 10_i) may be used to refer to non-specific instances of the class (members of the group / collection). DETAILED DESCRIPTION

[0059] Spatial audio refers to audio where the audio sources are distinctively localized, usually in three dimensions. Localization can be directional only, but can also be positional (direction and distance).

[0060] The captured audio scene captures audio sources at different locations in the audio space. It is captured as spatial audio content using multiple microphones (such as a microphone array).

[0061] The rendered audio scene renders audio sources at different positions in the rendered audio space. It is rendered from spatial audio content using multiple speakers (e.g., using a speaker setup or via headphones).

[0062] The headset may have head tracking capabilities. The audio (audio scene) rendered to the listener depends on the listener's position and / or orientation in the rendered audio space, and in some examples, this may track the listener's position and / or orientation in real space.

[0063] Channel-based audio (eg, nm surround sound, such as 5.1, 7.1, or 22.2 surround sound) or binaural audio may be used, or scene-based audio may be used, including spatial information about the sound field and audio sources.

[0064] The audio content may encode spatial audio as an audio object. Examples include, but are not limited to, MPEG-4 and MPEG SAOC. MPEG SAOC is an example of metadata-assisted spatial audio.

[0065] The audio content may encode spatial audio as audio objects in the form of moving virtual speakers.

[0066] The audio content may encode spatial audio as an audio signal with or without parametric side information or metadata. The audio signal may be, for example, first-order panoramic surround sound (Ambisonics, FOA) or its special instance B format, a high-order Ambisonics (HOA) signal or mid-side stereo. For such an audio signal, a synthesis of the audio signal and parametric metadata (if any) is used to synthesize the audio scene so as to create the desired spatial perception.

[0067] The parameterized metadata can be generated by different techniques. For example, spatial audio capture or directional audio coding (DirAC) can be used. Both capture the sound field and use parameterized metadata to represent it. The parameterized metadata may include, for example: a directional parameter indicating the direction of each frequency band; an energy split parameter indicating the diffuse-to-totalenergy ratio of each frequency band. Each time-frequency block can be regarded as an audio source, where the directional parameter controls the vector-based amplitude translation (VBAP) for the direct version and the energy split parameter controls the gain for the indirect (decorrelated) version.

[0068] In some embodiments, the parameterized audio metadata may relate to a Metadata Assisted Spatial Audio (MASA) format.

[0069] Metadata Assisted Spatial Audio (MASA) uses an audio signal and corresponding spatial metadata. The spatial metadata includes parameters that define the spatial aspects of the audio signal, and it may include, for example, direction and direct-to-total energy ratio in a frequency band. A MASA stream may be obtained, for example, by capturing spatial audio using a microphone of a suitable capture device. For example, a mobile device including multiple microphones may be configured to capture microphone signals, wherein a set of spatial metadata may be estimated based on the captured microphone signals. A MASA stream may also be obtained from other sources, such as specific spatial audio microphones (such as Ambisonics), studio mixes (e.g., 5.1 audio channel mixes), or other content through appropriate format conversion.

[0070] 3GPP IVAS (3GPP, Immersive Voice and Audio Services) and MPEG-I, which are currently under development and standardization, support new immersive voice and audio services, such as media reality.

[0071] The Immersive Voice and Audio Services (IVAS) codec has been designed to be suitable for use on communication networks (such as 3GPP4G / 5G networks), including use in such immersive services (e.g., immersive voice and audio for virtual reality (VR)). The audio codec handles the encoding, decoding, and rendering of speech, music, and general audio. In addition, it also supports channel-based audio and scene-based audio input, including spatial information about the sound field and audio source, i.e., metadata-assisted spatial audio (MASA) and object-based audio (or independent streams with metadata, ISM). The codec operates with low latency to enable conversational services. It also supports high error robustness under various transmission conditions. The IVAS codec supports rendering for speaker settings and binaural presentation, including head tracking binaural rendering.

[0072] In some but not necessarily all examples, amplitude panning techniques may be used to create or locate audio objects or sources. For example, known vector-based amplitude panning (VBAP) methods may be used to locate audio sources.

[0073] Audio objects can be repositioned by mixing the direct form of the object (attenuated and directionally filtered direct sound) with the indirect form of the object (e.g., localized directional early reflections and / or diffuse reverberation). If the audio source is louder and less reverberant, it appears closer, and if the audio source is quieter and more reverberant, it appears farther away.

[0074] In some but not necessarily all examples, the IVASMASA bitstream may be binaurally rendered using an integrated IVAS parameterized binaural renderer. In other examples, an external rendering such as a separate binaural renderer may be used.

[0075] Spatial audio capture can be performed using various means.

[0076] For example, a multi-microphone device such as a smartphone can use parametric capture to analyze and process the input signal and export a parametric spatial audio format or synthesize some other spatial audio format. Metadata Assisted Spatial Audio (MASA) is a parametric spatial audio format.

[0077] Ambisonics is a non-parametric spatial audio format. Spherical microphone arrays (such as Ambisonics microphone arrays) can be used to capture first-order Ambisonics (FOA) or higher-order Ambisonics (HOA) spatial audio. Often, FOA / HOA is also referred to as scene-based audio (SBA).

[0078] A common property of both the parametric spatial audio format (MASA) and scene-based audio (FOA / HOA or SBA) is that they represent an audio scene around a capture point (origin), which is usually understood as the center point of the microphone array or the center point of the corresponding transmission channel.

[0079] The 3GPP IVAS (Immersive Voice and Audio Services) codec is currently being standardized and will support both MASA and SBA input formats as well as, for example, stereo, independent streams with metadata (ISM), or audio objects.

[0080] Both the MASA and SBA formats have a default format orientation, i.e., the front of the audio scene corresponds to a certain direction in the captured scene. However, it may be desirable to render a different part of the audio scene as being in front of the listener. This may be referred to as scene orientation.

[0081] Figure 1A An example of an audio scene 20 before modification is illustrated. Figure 1B The audio scene 20 after modification is illustrated.

[0082] The audio scene 20 comprises one or more persons 2. In this example, it comprises two persons 2_1, 2_2.

[0083] Each person 2 is an audio source 4 or potential audio source. In this example, there is an audio source 4_1 associated with speech from person 2_1, and there is an audio source 4_2 associated with speech from person 2_2.

[0084] The audio scene 20 has a scene orientation 22 of the audio scene 20. In some, but not necessarily all examples, this scene orientation 22 is aligned with a notional point of view of a listener.

[0085] The audio scene 20 comprises a sector 30. The sector 30 of the audio scene 20 comprises one or more persons 2_1, 2_2 (and one or more associated audio sources 4_1, 4_2). The sector 30 has an orientation 32, ie a sector orientation.

[0086] If the audio scene 20 represents a captured audio scene, the spatial audio content captured by the microphones defines the direction (and possibly the position) of the input audio sources 4_1 , 4_2 relative to the origin 21 of the audio scene 20 .

[0087] If the audio scene 20 represents a rendered audio scene, the rendered spatial audio content defines the direction (and possibly the position) of the output audio sources 4_1 , 4_2 relative to the origin 21 of the audio scene 20 .

[0088] from Figure 1A It can be seen that before the audio scene modification, there is an angular offset 40 between the scene orientation 22 of the audio scene 20 and the sector orientation 32 of the sector 30. Figure 1B It can be seen that after the audio scene modification, the angular offset 40 between the scene orientation 22 of the audio scene 20 and the sector orientation 32 of the sector 30 is reduced.

[0089] In the illustrated example, the angular offset 40 is reduced to a zero offset, but other values ​​are possible.

[0090] To achieve this result:

[0091] (i) Identify a sector 30 of the audio scene 20. The sector 30 includes one or more persons 2_1, 2_2.

[0092] (ii) determining a modification to the spatial audio content for rendering the audio scene 20 to reduce an angular offset 40 between the scene orientation 22 of the audio scene 20 and the sector orientation 32 of the identified sector 30 .

[0093] This modification is used to render the audio scene 20, wherein the angular offset 40 between the scene orientation 22 and the sector orientation 32 is reduced.

[0094] In this example, the scene orientation 22 of the audio scene 20 is defined by the front of the scene direction for rendering the audio scene 20, and the sector orientation 32 of the identified sector 30 is defined by the centerline of the sector 30. The sector 30 has an angular extension 34, and the sector orientation 32 of the sector 30 is bisected by the centerline of the sector.

[0095] In this example, the audio scene 20 defined by the spatial audio content includes at least a first audio source 4_1 at a first orientation relative to the scene orientation 22 of the audio scene 20 and a second audio source 4_2 at a second orientation relative to the scene orientation 22 of the audio scene 20. The sum of a first angular offset between the first orientation and the scene orientation 22 and a second angular offset between the second orientation and the scene orientation 22 is Figure 1A and Figure 1B is reduced.

[0096] In this example, the first audio source 4_1 rotates around the origin 21, and the second audio source 4_2 also rotates around the origin. In this example, but not necessarily all examples, the first audio source 4_1 rotates a first amount around the origin 21 in a first scene, and the second audio source 4_2 also rotates a first amount around the origin in the first scene.

[0097] This rotation places the scene orientation 32 in the middle of the first and second orientations, so that the first audio source 4_1 is positioned on one side (the left side) of the scene orientation 22 and the second audio source 4_2 is positioned on the other side (the right side) of the scene orientation 22 .

[0098] The sum of a first angular offset between the first orientation and the scene orientation and a second angular offset between the second orientation and the scene is Figure 1A and Figure 1B is reduced.

[0099] In this example, but not necessarily in all examples, the angular offset 40 between the scene orientation 22 of the audio scene 20 and the sector orientation 32 of the identified sector 30 is a common rotation provided by the scene rotation of the audio scene about the origin 21 of the audio scene 20, such as Figure 1B as shown in .

[0100] The positional relationship between the audio sources 4 within the sector 30 is maintained and is not altered by this modification. The sector 30 retains its audio integrity. Instead, this modification causes the sector 30 to rotate about the origin 21.

[0101] Figure 2A and Figure 2B The diagram shows Figure 1A and Figure 1B The examples shown in FIG. 1 are similar to those shown in FIG. 1 , and the description provided above also relates to these figures. Figure 2A The audio scene 20 is shown before modification. Figure 2B The audio scene 20 is illustrated after modification. In this example, the audio scene 20 includes a plurality of persons 2 and a plurality of associated audio sources 4. Each person 2 is an audio source 4 or a potential audio source.

[0102] The audio scene 20 has a scene orientation 22 of the audio scene 20. In some, but not necessarily all examples, this scene orientation 22 coincides with a theoretical viewpoint of a listener.

[0103] The audio scene 20 comprises a sector-shaped portion 30. The sector-shaped portion 30 of the audio scene 20 is identified, which comprises a plurality of persons 2_1, 2_2, 2_3 (and associated audio sources 4).

[0104] from Figure 2A It can be seen that, before the modification, there is an angular offset 40 between the scene orientation 22 of the audio scene 20 and the sector orientation 32 of the sector 30. Figure 2B , after modification, the angular offset 40 between the scene orientation 22 of the audio scene 20 and the sector orientation 32 of the sector 30 is reduced. In the illustrated example, the angular offset 40 is reduced to zero offset, but other values ​​are possible.

[0105] To achieve this result:

[0106] (i) A sector-shaped portion 30 of the audio scene 20 is identified. The sector-shaped portion 30 comprises a plurality of persons 2. The sector-shaped portion 30 has an angular extension 34 depending on the spatial distribution of the persons 2 in the audio scene.

[0107] (ii) determining a modification to the spatial audio content for use in rendering the audio scene 20 to reduce an angular offset 40 between the scene orientation 22 of the rendered audio scene 20 and the sector orientation 32 of the identified sector 30 .

[0108] This modification is then used to render the audio scene 20, wherein the angular offset 40 between the scene orientation 22 and the sector orientation 32 is reduced.

[0109] The identification of the sector 30 of the audio scene 20 and the amount of rotation of the audio scene may be determined based on optimization of a cost function that favors a spaced distribution of the orientations of the persons 2 with respect to the scene orientations 22. In some examples, the cost function may also maintain a relative spatial distribution between the persons 2 in the audio scene 20.

[0110] Optimization of the cost function reduces the angular offset 40 between the scene orientation 22 of the audio scene 20 and the sector orientation 32 of the sector 30 and improves the balance of the audio scene. In the illustrated example, the audio source 4_i associated with the person 2_i is positioned further forward (determined by the scene orientation 22) and further to the left and right of the forward direction (determined by the scene orientation 22).

[0111] For example, if each audio source S_i has an audio source orientation (α_i, β_i) relative to the scene orientation 22, then minimizing

[0112]

[0113] Where, for example, F = sin((α_i)*sin(β_i) or

[0114] F=[sin((α_i)*sin(β_i)]^n, where n>1

[0115] In some examples, the cost function may be a weighted cost function, where the weights depend on characteristics of the audio sources S_i, such as the average energy or percentage of time the audio is produced.

[0116] For example, F = w_i*sin((α_i)*sin(β_i) or

[0117] F=w_i*[sin((α_i)*sin(β_i)]^n

[0118] Figure 3 An example of an apparatus 10 is illustrated, the apparatus comprising components for performing the following operations:

[0119] identifying a sector 30 of the audio scene 20, wherein the sector 30 includes one or more persons 2_1, 2_2; and

[0120] A modification of the spatial audio content is determined for use in rendering the audio scene 20 to reduce an angular offset 40 between the scene orientation 22 of the rendered audio scene 20 and the sector orientation 32 of the identified sector 30 .

[0121] In some, but not necessarily all, examples, the apparatus 10 additionally comprises a modification component configured to reduce an angular offset between the scene orientation 22 of the audio scene 20 and the sector orientation 32 of the identified sector 30 .

[0122] The means 16 for identifying a sector 30 of an audio scene comprising one or more persons 2 comprises processing means, for example a controller.

[0123] In some examples, the processing component 16 is configured to process the spatial audio content 11 captured by the microphone 12 from the audio scene 20 to identify a sector-shaped portion 30 of the audio scene that includes the speech audio source 4 .

[0124] In some examples, additionally or alternatively, processing component 16 is configured to process input 13 from one or more human presence sensors 14 for audio scene 20 that detect the physical presence of one or more people in audio scene 20 .

[0125] In some examples, human presence sensor 14 is a sensor that detects when a person is present at a particular location (eg, sitting in a particular seat).

[0126] In some examples, the human presence sensor 14 is a vehicle-mounted human presence sensor configured to detect the physical presence of a person in an audio scene, wherein the audio scene is an in-vehicle audio scene. The vehicle-mounted human presence sensor is configured to detect the physical presence of a person in a vehicle. In some examples, the vehicle-mounted human presence sensor includes a vehicle seat weight sensor 14_1 and / or a vehicle seat belt buckle activation sensor 14_2.

[0127] In some examples, human presence sensor 14 includes a camera and uses computer vision techniques to identify the presence of a person at a particular location.

[0128] Figure 4 A method 100 for adaptive spatial audio capture is illustrated, as previously described with respect to Figure 1A and Figure 1B as well as Figure 2A and Figure 2B The method may be executed by the device 10, for example.

[0129] The method comprises: identifying 104, 106, 108 a sector-shaped portion of an audio scene 20, wherein the sector-shaped portion comprises one or more persons 2; and

[0130] A modification of the spatial audio content is determined 114 for rendering the audio scene 20 to reduce an angular offset between the scene orientation 22 of the audio scene 20 and the sector orientation 32 of the identified sector 30 .

[0131] Only if it is significant enough (eg, greater than a threshold), may the modification be subsequently used.

[0132] In more detail, at block 102 an immersive voice call is initiated.

[0133] At block 112 , the method 100 includes obtaining codec negotiation information identifying a selected spatial audio encoding format.

[0134] In some but not necessarily all examples, codec negotiation information refers to, for example, Session Description Protocol (SDP) parameters that define, for example, encoder input formats, bandwidths, etc. supported by the sending device.

[0135] At block 116 , method 100 includes capturing spatial audio.

[0136] At block 118 , the method 100 includes forming a spatial audio input format suitable for the negotiated codec information.

[0137] At block 104 , method 100 includes obtaining a possible location of a person (a possible speaker).

[0138] At block 106 , the method 100 includes determining a sector of the activity based on the obtained position.

[0139] In some examples, based on determining that a sector of the audio scene actively (currently) includes one or more people, the sector is identified as an active sector of the audio scene. The active sector is an activity-presence sector.

[0140] In some examples, based on determining that the sector of the audio scene currently includes one or more people who are actively / actively (currently) speaking, the sector is identified as an active sector of the audio scene. The active sector is an activity-speaking sector.

[0141] In some examples (e.g., in a vehicle), a seat may be occupied by a person, but the person is not speaking or has not spoken recently. In this case, it can be determined that the sector associated with the seat is not active because, although it is an activity-presence sector (e.g., based on seat sensors), it is not an activity-speech sector (e.g., based on captured audio analysis).

[0142] In some examples, the components for identifying a sector of an audio scene that includes one or more persons are configured to identify the sector as an active sector and include components for processing audio content captured from the audio scene to identify a sector of the audio scene that actively includes a speech audio source.

[0143] In some examples, components for identifying a sector-shaped portion of an audio scene that includes one or more people are configured to identify the sector-shaped portion as an active sector-shaped portion and include components for receiving input from one or more human presence sensors that detect the active physical presence of one or more people in the sector-shaped portion of the audio scene.

[0144] In some examples, the component for identifying a sector of an audio scene that includes one or more people is configured to identify the sector as an active sector and includes one or more vehicle presence sensors configured to detect the active physical presence of one or more people in the sector of the audio scene, wherein the audio scene is an in-vehicle audio scene. In some examples, the one or more vehicle presence sensors include a vehicle seat weight sensor and / or a vehicle seat belt buckle activation sensor.

[0145] At block 112 , the method 100 includes forming the sector portions 30 of the overall activity based on the sector portions of the individual activities.

[0146] Thus, the method 100 comprises identifying 104, 106, 108 a sector 30 of an audio scene, wherein the sector comprises one or more persons.

[0147] At block 110 , the method includes determining audio scene orientation information based on the sector 30 of the overall activity.

[0148] At block 114, the method includes determining modification information based on the sector orientation 32 of the overall active sector 30. Block 114 determines the modification required to reduce the angular offset 40 between the scene orientation 22 of the audio scene 20 and the sector orientation 32 of the identified sector 30.

[0149] Thus, the method 100 comprises determining 114 a modification of the spatial audio content for rendering the audio scene to reduce the angular offset 40 between the scene orientation 22 of the audio scene and the sector orientation 32 of the identified sector 30 .

[0150] In one implementation, the determined modifications to the spatial audio content may be stored as a modification record of the modifications to be performed on the spatial audio content to reduce the difference between the orientation of the audio source and the scene orientation of the audio scene, so that the modifications can be performed elsewhere. The determined modifications and the spatial audio content are transmitted from the apparatus 10 so that the modification of the spatial audio scene can be performed downstream at or after the receiver after the transmission.

[0151] In another implementation, the spatial audio content is modified by the device 10 in the processing component 16 to reduce the difference between the sector orientation 32 of the sector 30 and the scene orientation 22 of the audio scene 20. The determined modification to the spatial audio content can be stored as a modification record of the modification performed on the spatial audio content to reduce the difference between the orientation of the audio source and the scene orientation of the audio scene, so that no modification is performed downstream at or after the receiver after transmission. The determined modification and the modified spatial audio content are sent to enable the use of the modified spatial audio scene, or, if necessary, to restore the original spatial audio scene before the modification.

[0152] In some examples, the modification determination may be substantially continuous, or, for example, periodic. Some smoothing of scene-oriented updates may be used. In some examples, method 100 is performed when a telecommunications session is initiated. In some examples, method 100 is performed when a new person speaks in an ongoing telecommunications session. In some examples, method 100 is performed when a new person joins an ongoing telecommunications session.

[0153] A telecommunication session enables the transmission or exchange of voice over a distance. An example is an immersive voice call.

[0154] It will be appreciated from the above that the apparatus 10 may include components for performing the following operations:

[0155] identifying an active sector of the audio scene, wherein the active sector includes at least one or more persons;

[0156] determining a modification to the spatial audio content for rendering the audio scene to reduce an angular offset between a scene orientation of the audio scene for rendering and a sector orientation of the identified sector of activity; and

[0157] Based on the determined modification, the spatial audio content is modified for rendering the audio scene to reduce an angular offset between a scene orientation of the rendered audio scene and a sector orientation of the identified sector of the activity in the rendered audio scene.

[0158] The method 100 can be dynamic and continuous so that changes to the active sectors are identified in real time and there are corresponding changes in the modification of the spatial audio that occur accordingly. Thus, active sectors can be dynamically added and / or removed based on the presence / activity of people.

[0159] Figure 5 An example of a system suitable for conducting a telecommunication session, such as an immersive voice call, is illustrated. In this example, the transmission is the transfer 44 of information from a capture system (eg, device 10) to a renderer system (eg, device 70).

[0160] The capture system (eg, apparatus 10) has been described in the previous figures.

[0161] In this example, the spatial audio content (modified or unmodified) and the determined modifications are transmitted by the capture system 10 to the rendering system 70 using the IVAS (Immersive Voice and Audio Services) codec.

[0162] The determined modifications may for example be encoded as scene directed update signaling, which is supported by the upcoming IVAS codec.

[0163] As previously described, the capture system 10 captures the audio scene 20 using the microphone 12. The capture system 10 also determines modifications to be performed (or modifications to be performed) on the spatial audio content to reduce the difference between the sector orientation 32 of the identified sector 30 including the person 2 and the scene orientation 22 of the captured audio scene 20.

[0164] A sector 30 of the audio scene 20 is identified. The sector 30 includes one or more persons 2_1, 2_2. A modification of the spatial audio content is determined for rendering the audio scene 20 to reduce an angular offset 40 between a scene orientation 22 of the rendered audio scene 20 and a sector orientation 32 of the identified sector 30.

[0165] The spatial audio content (modified or unmodified) and the determined modifications are encoded by an IVAS encoder 41. The resulting bitstream is packetized by an RTP (Real-time Transport Protocol) packetizer 42 for transmission over a wireless network link 44.

[0166] The rendering device 70 receives the data packets / packets transmitted over the wireless network link 44. The received data packets / packets are depacketized into frames by the RTP depacketizer 46. These frames are decoded by the IVAS decoder / renderer 48. The IVAS decoder / renderer 48 enables rendering of the modified audio scene 20 (which reduces the angular offset 40 between the original orientation 22 of the audio scene 20 and the original orientation 32 of the identified sector 30). The modification of the audio scene can be performed at the rendering device 70 using the received modifications.

[0167] The spatial audio data is encoded and rendered for playback on a particular system, for example, for binaural audio playback at headphones 52 .

[0168] Alternatively, the rendering may be implemented by an external renderer 50 (rather than the IVAS renderer), for example the manufacturer's own renderer.

[0169] The receiving user listens to the immersive voice call on their headphones. The user can, for example, choose between the original audio scene orientation or a modified audio scene orientation with improved spatial balance.

[0170] When scene orientation information is taken into account, a spatial audio scene with improved spatial balance is rendered to the receiving user, which improves the user experience by making it easier for the user to distinguish talker directions.

[0171] In some examples, a spatial audio capture environment may have more than one spatial audio capture array. A high-end vehicle may have multiple arrays. In this case, selecting between the first and second arrays may be based at least in part on obtaining possible speaker locations.

[0172] Figure 5 The different stages are illustrated using dashed lines. These stages can be performed in different devices or in the same device.

[0173] For example, audio scene capture by microphone 12 may be performed by an accessory device or by an onboard device. For example, processing 16 may be performed by an accessory device, a user's user equipment (UE), or by an onboard device. For example, IVAS encoding may be performed by a user's user equipment (UE) or by an onboard user equipment (UE).

[0174] For example, audio scene playback 53 can be performed by an accessory device (e.g., headphones) or by an in-vehicle device. For example, rendering 50 can be performed by an accessory device, a user's UE, or by an in-vehicle device. For example, IVAS decoding can be performed by a user's UE or by an in-vehicle UE.

[0175] In some examples, the spatial audio capture system 10 is used for a first user and the spatial audio rendering system 70 is used for a second user. In some examples, the spatial audio capture system 10 is used for the second user and the spatial audio rendering system 70 is used for the first user. In some examples, a single system (or device) can operate as the spatial audio capture system 10 and the spatial audio rendering system 70 for the first user, and a single system (or device) can operate as the spatial audio capture system 10 and the spatial audio rendering system 70 for the second user.

[0176] In at least some examples, microphone 12 for capturing spatial audio from an audio scene is fixed relative to audio scene 20. In this case, process 16 of determining modifications to the spatial audio content for rendering the audio scene to reduce the angular offset between scene orientation 20 of audio scene 20 and sector orientation 32 of identified sector 30 includes components for virtually reorienting the microphone.

[0177] During audio capture, it is not possible to control the position of person 2. However, in a controlled environment, the person may be expected to be stationary, e.g., the person may not be expected to move around the capture point origin 21. The possible talker positions are understandable and may be assumed to be substantially stationary relative to the capture point. Aspects of the audio scene 20 may be based in part on having information about where the (possible) talkers are (i.e., where person 2 is).

[0178] In the following examples, the spatial audio capture environment is a vehicle 60, in particular a cabin of a car. However, the described examples relate to any substantially stationary spatial audio capture setup, for example in a conference room, a living room setup, etc.

[0179] The microphone 12 used to capture spatial audio from the audio scene 20 is fixed relative to the vehicle 60 (in the interior of the vehicle cabin).

[0180] exist Fig. 6A and Figure 6B (audio capture) and Fig. 7A and Figure 7B In the example shown in (Audio Rendering), there is audio hardware, processing hardware, encoding hardware, and communication hardware. These hardware can be in the same or different devices 10, 70, for example, the audio hardware can be a car accessory or UE,

[0181] For example, the processing hardware could be an in-vehicle accessory or a UE.

[0182] For example, the encoding hardware could be an in-vehicle accessory or a UE.

[0183] For example, the communication hardware may be a UE,

[0184] In the example shown, microphone 12 is a vehicle-mounted accessory.

[0185] The processing hardware can be an in-vehicle accessory or a UE (on-board or user equipment).

[0186] The encoding hardware can be an in-vehicle accessory or UE (on-board or user)

[0187] The communication hardware can be a vehicle-mounted UE or a user UE (vehicle-mounted or user).

[0188] The user may be using a device such as a smartphone (UE), and the audio capture system integrated in the car may be considered an accessory available using any suitable wireless or wired connection.

[0189] A car may provide an integrated UE, for example, that includes the necessary components and functionality such as a 5G modem (or equivalent) for connection over the network.

[0190] In a further example, tethering may mean using primarily the capabilities of the car UE, but with the phone providing data transmission.

[0191] Spatial audio capture in a car can be enabled or improved by using an integrated microphone array. The spatial audio capture microphone array can be integrated, for example, into a rearview mirror assembly and / or in the center of the cabin / cab of the car. Preferably, the microphones are placed facing in the opposite direction of travel so that the speaker appears primarily in the front sector of the spatial audio scene 20, which can provide a more pleasant and natural listening experience.

[0192] Figure 8 The example of FIG. 9 illustrates the use of a sectored portion at a vehicle 60 for in-vehicle audio capture.

[0193] Audio capture comes from a microphone array 12 (not shown) at the interior rear view driver's mirror.

[0194] The cabin of the vehicle 60 is divided into different sectors 30_i, each sector being associated with a position of a person 2 (not shown).

[0195] The fan-shaped portion 30_1 is associated with a person located in the right front portion of the compartment of the vehicle 60. The fan-shaped portion 30_1 has a fan-shaped portion orientation (not shown) pointing to a seat at the right front of the compartment and having an angle extension 34_1. The fan-shaped portion 30_2 is associated with a person located in the right rear portion of the compartment of the vehicle 60. The fan-shaped portion 30_2 has a fan-shaped portion orientation (not shown) pointing to a seat at the right rear of the compartment and having an angle extension 34_2. The fan-shaped portion 30_3 is associated with a person located in the middle rear portion of the compartment of the vehicle 60. The fan-shaped portion 30_3 has a fan-shaped portion orientation (not shown) pointing to a seat at the middle rear of the compartment and having an angle extension 34_3. The fan-shaped portion 30_4 is associated with a person located in the left rear portion of the compartment of the vehicle 60. The fan-shaped portion 30_4 has a fan-shaped portion orientation (not shown) pointing to a seat at the left rear of the compartment and having an angle extension 34_4. The sector 30_5 is associated with a person (the driver in the left-hand drive vehicle) located in the left front part of the cabin of the vehicle 60. The sector 30_5 has a sector orientation (not shown) pointing to the seat at the left front of the cabin and having an angular extension 34_5.

[0196] In this example, but not necessarily all examples, the fan-shaped portions partially overlap. The fan-shaped portions 30_1, 30_3, 30_3 are continuous, and the fan-shaped portion 30_2 overlaps with the fan-shaped portions 30_1, 30_3, and the fan-shaped portion 30_4 overlaps with the fan-shaped portions 30_3, 30_5.

[0197] In this example, there are five sectors, but there may be a different number of sectors in different use cases. In this example, the sectors have a specific distribution, but there may be a different distribution of sectors in different use cases. For example, in the conference room example, the situation may be completely different. In the example, some or all of the sectors do not overlap.

[0198] A superset of the sectors typically covers only a portion of the 360 ​​degrees azimuth. The sectors may be understood as being azimuth only, e.g., in the case of planar spatial capture (e.g., planar FOA, a "3-component" spatial audio representation), or they may be understood as also covering the altitude component.

[0199] In some examples, the sector 30 is predetermined, corresponding to each seat position relative to the capture position.

[0200] In some examples, the determination of the activity direction / sector can be based at least in part on machine learning processing and / or additional data inputs. For example, the system can take into account who is present and who is the recipient or what the call is related to. For example, the sector 30 can be weighted unevenly when determining scene orientation information. Or, for example, the selection between the first and second microphone arrays can be based on the type of call, and therefore on the expected interested participants.

[0201] exist Figure 8 In the figure, person 2_1 is sitting in the seat at the left front of the compartment. Person 2_2 is sitting in the seat at the left rear of the compartment. Therefore, sector 30_5 is the sector in use. Therefore, sector 30_4 is the sector in use.

[0202] The fan-shaped parts 30_1, 20_2, 30_3, 30_4, and 30_5 are respectively Fig.9A , Fig. 9B , Fig. 9C , Fig.9D , Fig.9E Sector 30_5 is shaded to indicate that it is the sector in use ( Fig.9E ). Sector 30_4 is shaded to indicate that it is the sector in use ( Fig.9D ). The sectors 30_1, 30_2, 30_3 are not shaded, indicating that they are not sectors in use ( Fig.9A , Fig. 9B , Fig. 9C ).

[0203] As previously described, the capture system 10 uses the microphone 12 to capture the audio scene 20, for example, obtains the possible position of a person (a possible speaker) from the vehicle-mounted human presence sensor 14 (e.g., weight sensor and seat belt buckle activation detection), determines the active in-use sectors 30_5, 30_4 based on the obtained positions, and forms the overall active sector 30 based on the individual active in-use sectors 30_4, 30_5. Fig. 10A A sector (sector 30) of the overall activity is illustrated in FIG. The sector includes an audio source 4_1 associated with person 2_1 and an audio source 4_2 associated with person 2_2.

[0204] In some examples, the overall active sector (sector 30) includes active sector 30_5, 30_4 in use and any other sector except in the middle (between) of active sector 30_5, 30_4 in use.

[0205] As previously described, there may be alternative processes for determining the active or in-use sectors. For example, audio analysis may be used in conjunction with analysis of onboard human presence sensors and / or images captured from cameras.

[0206] The evaluation of active sectors and the modification of the spatial audio content can dynamically adapt and change, for example, when a person enters the vehicle or when a previously silent person in the vehicle speaks.

[0207] Fig. 10B It illustrates how the spatial audio rendering device 70 renders the original spatial audio content to the user. The audio sources 4_1 , 4_2 are rendered to the front and to the right and have a poor spatial separation / separation.

[0208] The capture system 10 determines modifications to be performed on the spatial audio content to reduce the difference between the sector orientation 32 of the identified sector 30 including the persons 4_1 , 4_2 and the scene orientation 22 of the captured audio scene 20 .

[0209] Fig. 11B It illustrates how the spatial audio rendering device 70 renders the modified spatial audio content to the user. The audio sources 4_1 , 4_2 are rendered to the front and center and with better spatial separation / separation.

[0210] The first person 2_1 (audio source 4_1) will be rendered to a listener on one side of the audio scene 20, and the second person 2_2 (audio source 4_2) will be rendered to a listener on the other side of the audio scene 20. In this example, the first person is the driver of the vehicle and the second person is a rear passenger in the vehicle.

[0211] Fig.11A The effective rotation of the audio scene 20 relative to the vehicle 60 resulting from this modification is shown.

[0212] refer to Fig. 12A In the example of , audio is captured from a microphone array 12 (not shown) at the center of the cabin of the vehicle 60. Person 2-1 is sitting in a seat at the left front of the cabin. Sector 30 is a sector in use.

[0213] As previously described, the capture system 10 uses the microphone 12 to capture the audio scene 20, obtains possible locations of people (possible speakers) from, for example, a vehicle-mounted human presence sensor, determines active in-use sectors 30 based on the obtained locations, and forms an overall active in-use sector 30 based on the individual active in-use sectors. Fig. 12A A sector (sector 30) of the overall activity is illustrated in FIG. The sector includes an audio source 4_1 associated with a person 2_1.

[0214] Fig. 12B The figure shows how the spatial audio rendering device 70 renders the original spatial audio content to the user before modification. The audio source 4_1 is rendered to the rear and to the right. This may cause inconvenience to the listener.

[0215] The capture system 10 determines modifications to be performed on the spatial audio content to reduce the difference between the sector orientation 32 of the identified sector 30 including the person 2_1 and the scene orientation 22 of the captured audio scene 20 .

[0216] Fig. 13B It illustrates how the spatial audio rendering device 70 renders modified spatial audio content to the user when central rendering is preferred. The audio source 4_1 is rendered to the front and center. Fig.13A The effective rotation of the audio scene 20 resulting from this modification is shown relative to the vehicle 60. In this example, the offset between the orientation of the audio source 4_1 and the scene orientation 22 (not shown) of the audio scene 20 has been reduced to zero.

[0217] Fig. 14B The example of illustrates how the spatial audio rendering device 70 renders modified spatial audio content to the user when central rendering is not preferred. The audio source 4_1 is rendered to the front and to the right. Fig.14A The effective rotation of the audio scene 20 relative to the vehicle 60 resulting from this modification is shown. In this example, the offset between the orientation of the audio source 4_1 and the scene orientation 22 (not shown) of the audio scene 20 has been reduced to a non-zero value. The audio source 4 has been moved closer to the front but is kept at its original side.

[0218] In some examples, the user (listener) determines whether the offset value is zero or non-zero. In some examples, the user (listener) determines the magnitude of the non-zero value.

[0219] Fig. 10A , Fig. 10B , Fig.11A , Fig. 11B , Fig. 12A , Fig. 12B , Fig.13A , Fig. 13B , Fig.14A , Fig. 14B An example of a spatial audio capture of a conversation originating from an in-vehicle environment and terminating in binaural headphone playback via headphones (which may or may not be head tracking headphones) is illustrated. Depending on the configuration of the speaker 2 in the vehicle 60, the default format orientation may provide a spatially unbalanced rendering. While this is a corrective rendering in itself, providing a modified spatial audio rendering with a more spatially balanced content representation allows for a more natural conversation. Improving the balance of the spatial scene rendering is particularly important for users who do not have head tracking capabilities (or other controls) or are in an environment where it is undesirable or impossible to change the listening orientation (e.g., rotate their head for a long period of time). Therefore, it is best to be able to automatically render the default scene to the user in a modified, spatially balanced manner. It may also be desirable to maintain the integrity of the audio scene (maintaining the relative positioning between the audio sources 4).

[0220] Fig.15A and Fig. 15C An example of rendering an audio scene 20 including audio sources using headphones 52 (eg binaural headphones) (without head tracking) is illustrated. The audio source 4 has a fixed orientation relative to the head and headphones 52 of the listener.

[0221] Fig.15A and Fig. 15B An example of using headphones 52 (e.g., binaural headphones) (with head tracking) to render an audio scene 20 including an audio source 4 is illustrated. The audio source 4 is rendered with a fixed orientation relative to the real world. The audio source 4 is rendered with an inverse rotation relative to the listener's head and the headphones 52.

[0222] When head tracking is performed, the perception is that the audio scene 20 remains stationary (fixed to the real world). For headphones 52, this is achieved by measuring the rotation of the listener's head and then counter-rotating the audio scene 20 by the same amount. A listener rotating his head to the right correspondingly "moves" all audio sources to the left in binaural perception. This may be referred to as, for example, head or user orientation.

[0223] In perspective-mediated virtual reality (e.g., first-person perspective-mediated reality), a user change to the position of a virtual listener within an audio scene changes the position of an audio source 4 relative to the virtual listener, which changes the virtual audio scene rendered to the user. A portion of the spatial audio content is selected by the user's current viewpoint (the virtual user's viewpoint). This portion of the spatial audio content is rendered to the user. The user can change the virtual user's viewpoint by changing their own viewpoint to understand different aspects of the audio space. In some examples, the change in the user's viewpoint is achieved by changing only the user's orientation, and in other examples, the change in the user's viewpoint is achieved by changing the user's orientation and / or the user's position. Therefore, the spatial audio content can support 3DoF, 3DoF+, and 6DoF.

[0224] Fig.16 An implementation of the IVAS decoder 48 comprising an integrated IVAS renderer at the rendering device 70 is illustrated.

[0225] The IVAS decoder 48 with integrated renderer provides means for re-orienting the audio scene for head tracking (among other scenarios, for example, via external orientation information 502).

[0226] The determined modifications described previously may be provided to the decoder / renderer 48 in the form of modification information as external orientation information 502. The external orientation information 502 input supports IVAS scene orientation update signaling.

[0227] In the illustrated example, there may be multiple modes of operation 503. A first mode corresponds to providing reference data 204 indicating a forward direction for rendering. A second mode may correspond to averaging the reference, e.g., a mechanism for removing slowly evolving orientation changes from head tracking data. A third mode may be a default mode or regular head tracking mode that does not apply any reference data or average head tracking or orientation data 206.

[0228] Head orientation flag 508 may be used to enable, disable, or freeze head tracking.

[0229] A head tracker process 510 determines the rotation of the audio scene based on the head orientation data 506 and if head tracking is enabled, the rotation is applied by a rotator 514 .

[0230] The output from the rotator 514 is rendered 50 as output audio, for example, binaural audio through headphones. The final directional data 212 from the rotator 514 may also be provided from the renderer 48 for use when required.

[0231] In addition to head tracking or orientation data 506 (if any), external orientation information 502 is provided to the renderer 48.

[0232] The rotator function 514 is separate from the head tracker process 510. This is because the external orientation data 502 is combined with the head tracker process 510 output by the orientation data combiner 516.

[0233] The directional data combiner 516 may also accept an external directional flag 509 as a command to enable, disable, or freeze the application of the external directional information 502 .

[0234] The external orientation 502 may change significantly at any update, so in some cases it may be desirable to smooth such changes. To this end, an interpolation function 518 may be provided. A new intermediate orientation based on a smooth transition from the initial orientation to the final orientation may be set for each frame. Therefore, the renderer 48 calculates the target orientation at each frame based on the current orientation, the external orientation 502, and the frame number.

[0235] Fig.17 An example of a controller 400 suitable for use in device 10 or device 70 is illustrated. An implementation of controller 400 may be a controller circuit. In some examples, controller 400 may be implemented solely in hardware, with certain aspects in software (including firmware), or may be a combination of hardware and software (including firmware).

[0236] like Fig.17 As shown in , the controller 400 can be implemented using instructions that implement hardware functions, for example, by using executable instructions of a computer program 406 in a general-purpose or special-purpose processor 402, which instructions can be stored on a computer-readable storage medium (disk, memory, etc.) for execution by such processor 402.

[0237] The processor 402 is configured to read from and write to the memory 404. The processor 402 may also include an output interface via which the processor 402 outputs data and / or commands and an input interface via which data and / or commands are input to the processor 402.

[0238] The memory 404 stores a computer program 406, which includes computer program instructions (computer program code) that control the operation of the device 10 when loaded into the processor 402. The computer program instructions of the computer program 406 provide the logic and routines that enable the device to perform the methods shown in the drawings. The processor 402 can load and execute the computer program 406 by reading the memory 404.

[0239] In at least some examples, the apparatus 10 includes:

[0240] at least one processor 402; and

[0241] at least one memory 404 comprising computer program code,

[0242] The at least one memory stores instructions that, when executed by the at least one processor 402, cause the apparatus to at least:

[0243] identifying a sector 30 of the audio scene 20, wherein the sector 30 includes one or more persons 2; and

[0244] A modification of the spatial audio content is determined for use in rendering the audio scene 20 to reduce an angular offset 40 between the scene orientation 22 of the rendered audio scene 20 and the sector orientation 32 of the identified sector 30 .

[0245] like Fig.18 As shown in , the computer program 406 may arrive at the apparatus 10 or the apparatus 70 via any suitable delivery mechanism 408. The delivery mechanism 408 may be, for example, a machine-readable medium, a computer-readable medium, a non-transitory computer-readable medium, a computer program product, a memory device, a recording medium such as a compact disk read-only memory (CD-ROM) or a digital versatile disk (DVD) or a solid-state memory, an article of manufacture including or tangibly embodying the computer program 406. The delivery mechanism may be a signal configured to reliably transmit the computer program 406. The apparatus 10 may propagate or send the computer program 406 as a computer data signal.

[0246] Computer program instructions for causing the apparatus 10 to at least perform the following operations or for at least performing the following operations:

[0247] identifying a sector 30 of the audio scene 20, wherein the sector 30 includes one or more persons 2; and

[0248] A modification of the spatial audio content is determined for use in rendering the audio scene 20 to reduce an angular offset 40 between the scene orientation 22 of the rendered audio scene 20 and the sector orientation 32 of the identified sector 30 .

[0249] The computer program instructions may be included in a computer program, a non-transitory computer readable medium, a computer program product, a machine readable medium. In some but not necessarily all examples, the computer program instructions may be distributed over more than one computer program.

[0250] Although memory 404 is shown as a single component / circuit, it may be implemented as one or more separate components / circuit, some or all of which may be integrated / removable and / or may provide permanent / semi-permanent / dynamic / cache storage.

[0251] Although processor 402 is shown as a single component / circuit, it may be implemented as one or more separate components / circuit, some or all of which may be integrated / removable.Processor 402 may be a single-core or multi-core processor.

[0252] References to "computer-readable storage medium," "computer program product," "tangibly embodied computer program," etc., or "controller," "computer," "processor," etc., should be understood to cover not only computers having different architectures such as single / multiple processor architectures and serial (von Neumann) / parallel architectures, but also special purpose circuits such as field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), signal processing devices, and other processing circuits. References to computer programs, instructions, code, etc. should be understood to cover software for a programmable processor, or firmware such as programmable content of a hardware device that may include instructions for a processor, or configuration settings for a fixed function device, gate array, or programmable logic device, etc.

[0253] As used in this application, the term "circuitry" may refer to one or more or all of the following:

[0254] (a) hardware circuit implementation only (such as analog and / or digital circuit implementation only);

[0255] (b) a combination of hardware circuitry and software such as (where applicable):

[0256] (i) a combination of analog and / or digital hardware circuits and software / firmware; and

[0257] (ii) any portion of a hardware processor with software (including a digital signal processor, software and memory that work together to enable a device such as a mobile phone or server to perform various functions); and

[0258] (c) Hardware circuits and / or processors, such as a microprocessor or portion of a microprocessor, that require software (e.g., firmware) to operate, but may not be present when the software is not required for operation.

[0259] This definition of "circuitry" applies to all uses of this term in this application, including in any claims. As another example, as used in this application, the term "circuitry" also covers an implementation of only hardware circuits or processors and their accompanying software and / or firmware. The term "circuitry" also covers (for example and if applicable to a particular claim element) a baseband integrated circuit for a mobile device, or a similar integrated circuit in a server, cellular network device, or other computing or network device.

[0260] The blocks shown in the accompanying drawings may represent steps in a method and / or code segments in a computer program 406. The illustration of a particular order of blocks does not imply a required or preferred order for the blocks, but the order and arrangement of the blocks may be changed. In addition, some blocks may be omitted.

[0261] If a structural feature has been described, it may be replaced by a component for performing one or more functions of the structural feature, whether the function or functions are explicitly or implicitly described.

[0262] According to the examples of the present disclosure, the device 10 can be provided in an electronic device (e.g., a mobile terminal). However, it should be understood that the mobile terminal is only used to illustrate the electronic device that will benefit from the implementation examples of the present disclosure, and therefore should not be considered to limit the scope of the present disclosure to this. Although in some implementation examples, the device can be provided in a mobile terminal, other types of electronic devices (such as but not limited to: mobile communication devices, handheld portable electronic devices, wearable computing devices, portable digital assistants (PDAs), pagers, mobile computers, desktop computers, televisions, gaming devices, laptops, cameras, video recorders, GPS devices, and other types of electronic systems) can easily adopt the examples of the present disclosure. In addition, the device can easily adopt the examples of the present disclosure, regardless of whether it is intended to provide mobility.

[0263] The term "comprising" as used in this document has an inclusive rather than exclusive meaning. That is, any expression "X comprises Y" means that X may comprise only one Y or may comprise more than one Y. If the exclusive meaning of "comprising" is intended, it will be made clear in the context by mentioning "comprising only one..." or using "consisting of..."

[0264] In this description, the words "connected", "coupled", and "in communication" and their derivatives mean operationally connected / coupled / coupled / in communication. It should be understood that any number or combination of intermediate components (including no intermediate components) may be present, i.e., to provide direct or indirect connection / coupling / coupling / in communication. Any such intermediate components may include hardware and / or software components.

[0265] As used herein, the term "determined" (and grammatical variations thereof) may include, but are not limited to, calculating, estimating, processing, deriving, measuring, investigating, identifying, searching (e.g., searching in a table, database, or other data structure), ascertaining, etc. In addition, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), obtaining, etc. In addition, "determining" may include solving, selecting, choosing, establishing, etc.

[0266] Reference has been made to various examples in this description. The description of features or functions for an example indicates that these features or functions are present in the example. Whether explicitly stated or not, the use of the term "example" or "for example" or "may" or "can" in the text indicates that such features or functions are present in at least the described example, whether or not described as an example, and such features or functions may, but not necessarily, be present in some or all other examples. Therefore, "example", "for example" or "may" or "can" refers to a specific instance in a class of examples. The properties of an instance may be only the properties of the instance or the properties of the class of instances or the properties of a subclass of the class of instances including some but not all of the class of instances. Therefore, it is implicitly disclosed that features described for one example but not for another example can be used for other examples as part of a working combination, but are not necessarily used for other examples.

[0267] Although the examples have been described in the preceding paragraphs with reference to various examples, it should be appreciated that modifications to the examples given can be made without departing from the scope of the claims.

[0268] Features described in the preceding description may be used in combinations other than the combinations explicitly described above.

[0269] Although functions have been described with reference to certain features, those functions may be performable by other features whether described or not.

[0270] Although features have been described with reference to certain examples, those features may also be present in other examples whether described or not.

[0271] As used herein, the terms "a", "an", or "the" have an inclusive, rather than exclusive, meaning. That is, any reference to "X includes a", "an", or "the" indicates "X may include only one Y" or "X may include more than one Y", unless the context clearly indicates otherwise. If the exclusive meaning of "a", "an", or "the" is intended, it will be clearly stated in the context. In some cases, "at least one" or "one or more" may be used to emphasize the inclusive meaning, but the absence of these terms should not be construed as implying any non-exclusive meaning.

[0272] The presence of a feature (or combination of features) in a claim is a reference to the feature (or combination of features) itself, and is also a reference to features that achieve substantially the same technical effect (equivalent features). Equivalent features include, for example, features that are variants and achieve substantially the same result in substantially the same way. Equivalent features include, for example, features that perform substantially the same function in substantially the same way to achieve substantially the same result.

[0273] Reference has been made in this specification to various examples using adjectives or adjective phrases to describe characteristics of the examples. Such description of characteristics with respect to the examples means that the characteristics exist in some examples exactly as described, and in other examples substantially as described.

[0274] The above description describes some examples of the present disclosure, however, those of ordinary skill in the art will be aware of possible alternative structures and method features that provide equivalent functionality to the specific examples of such structures and features described above, and have been omitted from the above description for the sake of brevity and clarity. Nevertheless, the above description should be understood to implicitly include reference to such alternative structures and method features that provide equivalent functionality, unless such alternative structures or method features are explicitly excluded in the above description of examples of the present disclosure.

[0275] While an attempt has been made in the foregoing description to point out those features regarded as important, it should be understood that the applicant may seek protection via the claims for any patentable feature or combination of features previously referenced herein and / or shown in the drawings, whether emphasized or not.

Claims

1. An apparatus comprising components for performing the following operations: Identify a sector of the audio scene where The sector-shaped portion includes one or more persons and the sector-shaped portion has an angular extension that does not extend beyond the spatial distribution of persons in the audio scene; as well as A modification of the spatial audio content is determined for rendering the audio scene to reduce an angular offset between a scene orientation of the audio scene for rendering and a sector orientation of the identified sector.

2. The device according to claim 1, wherein: The scene orientation of the audio scene for rendering is defined by a front direction for rendering the audio scene, and the sector orientation of the identified sector is defined by a midline of the sector.

3. A device according to any one of the preceding claims, wherein: The means for identifying a sector of an audio scene that includes one or more persons is configured to identify the sector as an active sector and includes means for processing audio content captured from the audio scene to identify a sector of the audio scene that actively includes a speech audio source.

4. A device according to any one of the preceding claims, wherein: A component for identifying a sector of an audio scene that includes one or more persons is configured to identify the sector as an active sector and includes a component for receiving input from one or more human presence sensors for the audio scene, the one or more human presence sensors detecting the active physical presence of one or more persons in the sector of the audio scene.

5. A device according to any one of the preceding claims, wherein: A component for identifying a sector-shaped portion of an audio scene including one or more persons is configured to identify the sector-shaped portion as an active sector-shaped portion, and includes one or more vehicle-mounted human presence sensors, wherein the one or more vehicle-mounted human presence sensors are configured to detect the active physical presence of one or more persons in the sector-shaped portion of the audio scene, wherein the audio scene is an in-vehicle audio scene.

6. The device according to claim 5, wherein: The one or more vehicle occupant presence sensors include a vehicle seat weight sensor and / or a vehicle seat belt buckle activation sensor.

7. A device according to any one of the preceding claims, wherein: Means for determining a modification to the spatial audio content for rendering the audio scene to reduce an angular offset between a scene orientation of the audio scene and a sector orientation of the identified sector comprises means for rotating the audio scene.

8. The device according to any one of the preceding claims, comprising: A modification component, the modification component is used to modify the spatial audio content for rendering the audio scene, wherein the spatial audio content includes at least a first audio source at a first orientation relative to the scene orientation of the audio scene and a second audio source at a second orientation relative to the scene orientation of the audio scene, wherein the modification component is configured to: reduce an angular offset between the first orientation and the scene orientation of the audio scene, and an angular offset between the second orientation and the scene orientation of the audio scene.

9. The device according to claim 8, wherein: The modification component is configured to modify the spatial audio content such that the scene orientation of the audio scene is intermediate between the first orientation and the second orientation.

10. The device according to any one of the preceding claims, wherein: A component for determining a modification of spatial audio content for rendering the audio scene to reduce the angular offset between the scene orientation of the audio scene and the sector-shaped portion orientation of the identified sector-shaped portion includes a modification component for modifying the spatial audio content for rendering the audio scene to optimize a cost function for an interval distribution of orientations favorable to a person with respect to the scene orientation of the audio scene.

11. The device according to any one of the preceding claims, wherein: Means for determining a modification to the spatial audio content for rendering the audio scene to reduce an angular offset between a scene orientation of the audio scene and a sector orientation of the identified sector comprises: means for determining a modification to the spatial audio content for rendering the audio scene to reduce to zero an angular offset between the scene orientation of the audio scene and the sector orientation of the identified sector; or Means for determining a modification to the spatial audio content for rendering the audio scene to reduce an angular offset between the scene orientation of the audio scene and the sector orientation of the identified sector to a non-zero value.

12. The device according to any one of the preceding claims, wherein: A microphone for capturing the spatial audio from an audio scene is fixed relative to the audio scene, and wherein the means for determining modifications to the spatial audio content for rendering the audio scene to reduce the angular offset between the scene orientation of the audio scene and the sector orientation of the identified sector comprises means for virtually reorienting the microphone.

13. An apparatus according to any of the preceding claims, configured to transmit a modification record of the modification of the spatial audio content to reduce the difference between the orientation of the audio source and the scene orientation of the audio scene, so that the modification can be performed or not performed.

14. A method comprising: identifying a sector-shaped portion of an audio scene, wherein the sector-shaped portion includes one or more persons and the sector-shaped portion has an angular extension that does not extend beyond a spatial distribution of persons in the audio scene; and A modification of the spatial audio content is determined for rendering the audio scene to reduce an angular offset between a scene orientation of the audio scene for rendering and a sector orientation of the identified sector.

15. A computer program comprising instructions which, when executed by one or more processors, cause: Identify a sector of the audio scene where The sector-shaped portion includes one or more persons and the sector-shaped portion has an angular extension that does not extend beyond the spatial distribution of persons in the audio scene; as well as A modification of the spatial audio content is determined for rendering the audio scene to reduce an angular offset between a scene orientation of the audio scene for rendering and a sector orientation of the identified sector.

Images