Device, method and computer program for enabling remote users to access mediated reality content

Abstract

An apparatus includes means for simultaneously controlling content presented by a handheld portable device and content presented by a spatial audio device; and in response to an action by the user, providing presentation of a first portion of the spatial audio content instead of a second portion to a user via the handheld portable device instead of the spatial audio device.

Description

Technical Field

[0001] Embodiments of the present disclosure are directed to enabling access to mediated reality content by remote users. Background Art

[0002] When mediated reality is presented to a user, the user experiences, for example visually and / or auditorily, a fully or partially artificial environment, such as a virtual scene at least partially presented to the user by the device. The virtual scene is determined by the viewpoint of the virtual user in the virtual space.

[0003] Augmented reality is a form of mediated reality in which a user experiences an environment that is partially artificial as a virtual scene, which includes a real scene of a physical real environment supplemented by one or more visual or audio elements presented to the user by a device, such as a real visual scene and / or a real sound scene. The term augmented reality means mixed reality or hybrid reality. Virtual reality is a form of mediated reality in which a user experiences a completely artificial environment, such as a virtual scene presented to the user by a device.

[0004] First person perspective mediation applied to mediated reality means that the user's real viewpoint (location and / or orientation) determines the viewpoint (location and / or orientation) within the virtual space of the virtual user.

[0005] When the mediated reality is presented to the user as a sound scene, a three-dimensional sound field can be created. Summary of the invention

[0006] According to various but not all embodiments, there is provided an apparatus comprising means for:

[0007] Simultaneously controlling content rendered by a handheld portable device and content rendered by a spatial audio device; and

[0008] In response to the user's action, presentation of the first portion of the spatial audio content, but not the second portion, is provided to the user via the handheld portable device instead of the spatial audio device.

[0009] In some but not necessarily all examples, the apparatus includes means for causing, prior to an action by a user, simultaneous presentation of the first portion of the spatial audio content and the second portion of the spatial audio content via the spatial audio device.

[0010] In some but not necessarily all examples, the apparatus includes means for determining, in response to a user action, a first portion of the spatial audio content and a second portion of the spatial audio content.

[0011] In some but not necessarily all examples, the apparatus includes means for causing, in response to a user action, simultaneous presentation to a user of a first portion of the spatial audio content via a handheld portable device and a second portion of the spatial audio content via a spatial audio device.

[0012] In some but not necessarily all examples, the spatial audio content is received by the device from a communication network as a downlink communication channel from a remote person, and wherein the first portion of the spatial audio content may include speech of the remote person.

[0013] In some but not necessarily all examples, the apparatus includes means for causing, in response to a user action, presentation of a first portion of the spatial audio content via one or more speakers of the handheld portable device.

[0014] In some but not necessarily all examples, the apparatus includes means for causing simultaneous presentation of a first portion of the spatial audio content via the handheld portable device and a second portion of the spatial audio content via a speaker of the spatial audio device as two-channel audio.

[0015] In some but not necessarily all examples, the apparatus includes components for communicating with the handheld portable device and / or the spatial audio device to determine when a user's action has occurred, wherein the user's action includes movement of the handheld portable device toward the user's head.

[0016] In some but not necessarily all examples of the first embodiment, the apparatus includes means for causing, in response to an action by a user, simultaneous presentation of a first portion of the spatial audio content to the user via a handheld portable device and presentation of a second portion of the spatial audio content via a spatial audio device.

[0017] In some, but not necessarily all, examples of the first embodiment, the spatial audio scene defined by the second portion of the spatial audio content is modified depending on the position of the handheld portable device.

[0018] In some but not necessarily all examples of the first embodiment, one or more audio objects that at least partially define the spatial audio scene are moved depending on the position of the handheld portable device to avoid the audio objects and the first portion of the spatial audio content determined by the position of the handheld portable device being located in the same place.

[0019] In some but not necessarily all examples of the first embodiment, a first portion of the spatial audio content is associated with a first speech associated with a first ear of a user at which the handheld portable device is positioned, and wherein the spatial audio content is associated with at least a different second speech, and wherein a spatial audio scene defined by a second portion of the spatial audio content is modified to place the second speech at a second ear of the user opposite the first ear.

[0020] In some but not necessarily all examples, the apparatus includes means for editing the second portion of the spatial audio content in response to movement of the handheld portable device.

[0021] In some but not necessarily all examples, the first portion of the spatial audio content is related to an incoming call and the second portion of the spatial content is related to at least content that is to continue to be presented after the incoming call is received.

[0022] In some but not necessarily all examples of the second embodiment, the apparatus includes means for enabling selection of the first portion of the spatial audio content by the user using a first-person perspective mediated reality.

[0023] In some but not necessarily all examples of the second embodiment, the first portion of the spatial audio content is an audio object selected by a direction of user attention, or an audio scene selected by a direction of user attention.

[0024] In some but not necessarily all examples of the second embodiment, the first portion of the spatial audio content is a first moving audio object, wherein the first portion of the audio content is presented as if the first audio object is stationary.

[0025] In some but not necessarily all examples, the apparatus includes means for implementing the adapting of the second portion of the spatial audio content using the first-person perspective mediated reality following a user action while using at least the spatial audio device.

[0026] In some but not necessarily all examples, the apparatus includes means for, following the user action, adapting a second portion of the spatial audio content according to the first-person perspective mediated reality, and presenting the second portion of the spatial audio content via at least the spatial audio device.

[0027] In some, but not necessarily all, examples, the apparatus includes means for: enabling adaptation of a second portion of the spatial audio content according to the first-person perspective mediated reality following the user action, and presenting the second portion of the spatial audio content via at least the spatial audio device;

[0028] Means for: after a subsequent user action, disabling adaptation of a second portion of the spatial audio content mediated reality according to the first-person perspective, and presenting the second portion of the spatial audio content according to the fixed first-person perspective via the spatial audio device;

[0029] In some, but not necessarily all, examples, the handheld portable device includes a display and is configured to serve as or house the apparatus.

[0030] According to various but not necessarily all embodiments, there is provided a method comprising:

[0031] Simultaneously controlling content rendered by a handheld portable device and content rendered by a spatial audio device; and

[0032] In response to the user's action, presentation of the first portion, but not the second portion, of the spatial audio content is provided to the user via the handheld portable device instead of the spatial audio device.

[0033] According to various but not necessarily all embodiments, there is provided a computer program which, when executed on one or more processors, implements:

[0034] Simultaneously controlling content rendered by a handheld portable device and content rendered by a spatial audio device; and

[0035] In response to the user's action, presentation of the first portion, but not the second portion, of the spatial audio content is provided to the user via the handheld portable device instead of the spatial audio device.

[0036] According to various but not all embodiments, there is provided an apparatus comprising:

[0037] at least one processor; and

[0038] at least one memory including computer program code,

[0039] The at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least perform:

[0040] Simultaneously controlling content rendered by a handheld portable device and content rendered by a spatial audio device; and

[0041] In response to the user's action, presentation of the first portion, but not the second portion, of the spatial audio content is provided to the user via the handheld portable device instead of the spatial audio device.

[0042] According to various but not all embodiments, examples are provided as claimed in the following claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Some example embodiments will now be described with reference to the accompanying drawings, in which:

[0044] Figure 1A , Figure 1B , Figure 1C , Figure 1D , Figure 1E and Figure 1F An example embodiment of the subject matter described herein is shown;

[0045] Figure 2 Another example embodiment of the subject matter described herein is shown;

[0046] Figure 3A , Figure 3B , Figure 3C Another example embodiment of the subject matter described herein is shown;

[0047] Figure 4A , Figure 4B , Figure 4C Another example embodiment of the subject matter described herein is shown;

[0048] Figure 5A , Figure 5B , Figure 5C Another example embodiment of the subject matter described herein is shown;

[0049] Fig. 6A , Figure 6B , Figure 6C Another example embodiment of the subject matter described herein is shown;

[0050] Fig. 7A , Figure 7B , Figure 7C Another example embodiment of the subject matter described herein is shown;

[0051] Figure 8 Another example embodiment of the subject matter described herein is shown;

[0052] Fig.9A , Fig. 9B , Fig. 9C , Fig.9D , Fig.9E Another example embodiment of the subject matter described herein is shown;

[0053] Fig. 10A , Fig. 10B , Fig. 10C , Fig. 10D Another example embodiment of the subject matter described herein is shown;

[0054] Fig.11A , Fig. 11B Another example embodiment of the subject matter described herein is shown; and

[0055] Fig. 12A , Fig. 12B Another example embodiment of the subject matter described herein is shown.

[0056] definition

[0057] An “artificial environment” can be something that has been recorded or generated.

[0058] A "virtual visual space" is a fully or partially artificial environment that can be viewed, which may be three-dimensional.

[0059] A "virtual visual scene" refers to a representation of a virtual visual space as viewed from a particular viewpoint (position) within the virtual visual space.

[0060] A "virtual visual object" is a visible virtual object within a virtual visual scene.

[0061] "Sound space" (or "virtual sound space") refers to the arrangement of sound sources in three-dimensional space. Sound space can be defined with respect to recorded sound (recorded sound space) and with respect to rendered sound (rendered sound space).

[0062] A "sound scene" (or "virtual sound scene") refers to a representation of a sound space as heard from a particular viewpoint (position) within the sound space.

[0063] "Sound Object" refers to a sound source that can be located within the sound space. In contrast to sound sources associated with objects in the virtual visual space, source sound objects represent sound sources within the sound space. Recorded sound objects represent sounds recorded at a specific microphone or location. Rendered sound objects represent sounds rendered from a specific location.

[0064] "Virtual space" may mean a virtual visual space, a sound space, or a combination of a virtual visual space and a corresponding sound space. In some examples, the virtual space may extend to 360° horizontally and 180° vertically.

[0065] “Virtual scene” may mean a virtual visual scene, mean a sound scene, or mean a combination of a virtual visual scene and a corresponding sound scene.

[0066] A "virtual object" is an object within a virtual scene, which can be an augmented virtual object (e.g., a computer-generated virtual object), or it can also be an image of a real object in a live or recorded real space. It can be a sound object and / or a virtual visual object.

[0067] A "virtual location" is a position within a virtual space. It may be defined using a virtual location and / or a virtual orientation. It may be thought of as a movable "viewpoint".

[0068] "Corresponding" or "corresponding" when used with respect to an acoustic space and a virtual visual space means that the acoustic space and the virtual visual space are aligned in time and space, ie, they are the same space at the same time.

[0069] When used with respect to a sound scene and a virtual visual scene (or visual scene), "corresponding" or "corresponding" means that the sound space and the virtual visual space (or visual scene) are corresponding, and that the nominal (virtual) listeners whose viewpoints define the sound scene and the nominal (virtual) viewers whose viewpoints define the virtual visual scene (or visual scene) are in the same place and orientation, i.e., they have the same viewpoint (same virtual position).

[0070] "Real space" (or "physical space") refers to a real environment, which may be three-dimensional.

[0071] A "real scene" refers to a representation of real space from a specific viewpoint (position) within the real space.

[0072] A "real visual scene" refers to a visual representation of real space as viewed from a particular real viewpoint (position) within the real space.

[0073] "Mediated reality" in this document refers to a fully or partially artificial environment (virtual space) that a user experiences, for example, visually and / or auditorily, as a virtual scene at least partially presented to the user by a device. The virtual scene is determined by a viewpoint (virtual position) within the virtual space. Displaying the virtual scene means providing the virtual visual scene in a form that the user can perceive.

[0074] "Augmented reality" in this document refers to a form of mediated reality in which a user experiences an artificial environment (virtual space) as part of a virtual scene that includes a realistic scene of a physical real environment (real space) supplemented by one or more visual or audio elements presented to the user by a device, such as a real visual scene. The term augmented reality means mixed reality or hybrid reality, and does not necessarily imply a degree of virtuality (as opposed to real) or intermediate degrees;

[0075] "Virtual Reality" in this document refers to a form of mediated reality in which a user experiences a completely artificial environment (virtual visual space) as a virtual scene displayed to the user by a device;

[0076] "Virtual content" is content, in addition to real content from the real scene (if any), that enables mediated reality by, for example, providing one or more augmented virtual objects.

[0077] "Mediated reality content" is virtual content that enables a user to experience a fully or partially artificial environment (virtual space) as a virtual scene, for example visually and / or auditorily. Mediated reality content can include interactive content such as video games or non-interactive content such as sports videos.

[0078] "Augmented reality content" is a form of mediated reality content that enables a user to experience an artificial environment (virtual space) as part of a virtual scene, for example visually and / or auditorily. Augmented reality content may include interactive content such as video games or non-interactive content such as sports videos.

[0079] "Virtual reality content" is a form of mediated reality content that enables a user to experience a completely artificial environment (virtual space) as a virtual scene, for example visually and / or auditorily. Virtual reality content may include interactive content such as video games or non-interactive content such as sports videos.

[0080] "Viewpoint mediation" as applied to mediated reality, augmented reality, or virtual reality means that user actions determine the viewpoint (virtual position) in the virtual space, thereby changing the virtual scene;

[0081] “First-person perspective mediation” as applied to mediated reality, augmented reality, or virtual reality means perspective mediation with the following additional constraints: the user’s real viewpoint (location and / or orientation) determines the virtual user’s viewpoint (virtual position) within the virtual space;

[0082] “Third-person perspective mediation” as applied to mediated reality, augmented reality, or virtual reality means perspective mediation with the following additional constraints: the user’s real viewpoint does not determine the viewpoint (virtual position) within the virtual space;

[0083] “User interaction” as applied to mediated reality, augmented reality, or virtual reality means that user actions at least partially determine what happens in the virtual space;

[0084] “Display” means to provide in a form that is visually perceived (viewed) by a user.

[0085] "Rendering" means providing in a user-perceived form.

[0086] A "virtual user" defines a viewpoint (virtual position, location and / or orientation) used to generate a perspective-mediated sound scene and / or visual scene in a virtual space. A virtual user can be a nominal listener and / or a nominal viewer.

[0087] The “nominal listener” defines the viewpoint (virtual position, location, and / or orientation) used to generate the perspective-mediated sound scene in virtual space, regardless of whether the user is actually listening.

[0088] The "nominal viewer" defines the viewpoint (virtual position, location, and / or orientation) used to generate a perspective-mediated visual scene in a virtual space, whether or not a user is actually looking.

[0089] Three degrees of freedom (3DoF) describes a mediated reality where the virtual position is determined only by orientation (e.g., three degrees of three-dimensional orientation). Examples of three degrees of three-dimensional orientation are pitch, roll, and yaw. With first-person perspective mediated reality 3DoF, only the user's orientation determines the virtual position.

[0090] Six degrees of freedom (6DoF) describes a mediated reality in which a virtual position is determined by both an orientation (e.g., three degrees of three-dimensional orientation) and a location (e.g., three degrees of three-dimensional location). An example of three degrees of three-dimensional orientation is pitch, roll, and yaw. An example of three degrees of three-dimensional location is a three-dimensional coordinate in Euclidean space spanned by orthogonal axes such as the left-to-right (x), front-to-back (y), and down-to-up (z) axes. With respect to first-person perspective mediated reality 6DoF, the user's orientation and the user's location in real space determine the virtual position. With respect to third-person perspective mediated reality 6DoF, the user's location in real space does not determine the virtual position. The user's orientation in real space may or may not determine the virtual position.

[0091] Three degrees of freedom "plus" (3DoF+) describes an example of six degrees of freedom where changes in location (e.g., three degrees of three-dimensional location) are relative to changes in the user's location, which may be due to changes in the posture of the user's head and / or body, and do not involve translation of the user in real space by, for example, walking. DETAILED DESCRIPTION

[0092] Figure 1A , Figure 1B , Figure 1C , Figure 1D , Figure 1E and Figure 1F A first person perspective mediated reality is shown. In this context, mediated reality means presenting a mediated reality for the purpose of achieving a mediated reality (e.g., augmented reality or virtual reality) for a remote user. It may or may not be user interactive. The mediated reality may support 3DoF, 3DoF+, or 6DoF. Alternatively, the mediated reality may support 3DoF or 6DoF without supporting 3DoF+.

[0093] Figure 1A , Figure 1C and Figure 1E The illustration shows a real space 50, an acoustic space 60 and a visual space 62 at a first time. There is a correspondence between the acoustic space 60 and the virtual visual space 62. A user 51 in the real space 50 has a point of view (position) 54 defined by a location 52 and an orientation 53. The location is a three-dimensional location and the orientation is a three-dimensional orientation.

[0094] In the example of 3DoF mediated reality, the user's real viewpoint 54 (orientation) determines the viewpoint 74 (virtual position) within the virtual space of the virtual user. The orientation 53 of the user 51 controls the virtual orientation 73 of the virtual user 71. There is a corresponding relationship between the orientation 53 and the virtual orientation 73, so that a change in the orientation 53 produces the same change in the virtual orientation 73.

[0095] The combination of the virtual orientation 73 and the virtual field of view 78 of the virtual user 71 defines a virtual visual scene 75 within the virtual visual space 62. In some examples, it may additionally or alternatively define a virtual sound scene 76.

[0096] Virtual visual scene 75 is the portion of virtual visual space 62 that is displayed to the user. Virtual sound scene 76 is the portion of virtual sound space 60 that is presented to the user. Virtual sound space 60 and virtual visual space 62 correspond in that locations within virtual sound space 60 have equivalent locations within virtual visual space 62. In 3DoF mediated reality, changes in location 52 of user 51 do not change virtual location 72 or virtual orientation 73 of virtual user 71.

[0097] In the example of 6DoF mediated reality, the user's real viewpoint 54 (location and / or orientation) determines the viewpoint 74 (virtual position) within the virtual space of the virtual user 71. This is as described for 3DoF, with the addition that it is possible to change the rendered virtual sound scene 76 and the displayed virtual visual scene 75 by movement of the location 52 of the user 51. For example, there may be a mapping between the location 52 of the user 51 and the virtual location 72 of the virtual user 71. Changes in the location 52 of the user 51 produce corresponding changes in the virtual location 72 of the virtual user 71. Changes in the virtual location 72 of the virtual user 71 change the rendered sound scene 76 and change the rendered visual scene 75.

[0098] This can be obtained from Figure 1B , Figure 1D and Figure 1F In the figure, it is understood that these figures illustrate the user 51 in the presented sound scene 76 ( Figure 1D ) and the presented visual scene 75( Figure 1F) is the consequence of a change in location 52 and orientation 53 on the user's device. The change in location may result from a change in the user's posture and / or translation of the user by walking or otherwise.

[0099] Depending on the implementation, the first-person perspective mediated reality may control only the virtual sound scene 76 , the virtual visual scene 75 , or both the virtual sound scene 76 and the virtual visual scene 75 .

[0100] In some cases, for example, when a sound scene is presented to a listener through a head-mounted audio output device (e.g., headphones using binaural audio encoding), it may be desirable for a portion of the presented sound space to remain fixed in real space as the listener turns his head in space. This means that the presented sound space needs to rotate by the same amount relative to the audio output device in the opposite sense to the rotation of the head. The orientation of the portion of the presented sound space tracks the rotation of the listener's head so that the orientation of the presented sound space remains fixed in space and does not move with the listener's head.

[0101] Sounds that are "locked" to the real world can be called "diegetic sounds".

[0102] Sounds that are "locked" to the user's head may be referred to as non-diegetic sounds.

[0103] The presentation of the virtual sound scene 76 may also be described as providing spatial audio or providing immersive audio. The virtual sound scene 76 includes one or more sound sources at different locations in the sound space 60. The audio presented to the user depends on the relative position of the virtual user and the location of the sound source. Perspective-mediated virtual reality, such as first-person perspective-mediated reality, enables the user 51 to change the position of the virtual user 71 within the sound space 60, thereby changing the position of the sound source relative to the virtual user, which changes the virtual sound scene 76 presented to the user 51.

[0104] Channel-based audio may be used, such as n, m surround sound (eg 5.1, 7.1 or 22.2 surround sound) or binaural audio, or scene-based audio may be used, including spatial information about the sound field and sound sources.

[0105] The audio content may encode spatial audio as an audio object. Examples include, but are not limited to, MPEG-4 and MPEG SAOC. MPEG SAOC is an example of metadata-assisted spatial audio.

[0106] The audio content may encode spatial audio as audio objects in the form of moving virtual speakers.

[0107] The audio content may encode spatial audio as an audio signal with parametric side information or metadata. The audio signal may be, for example, first order Ambisonics (FOA) or its special case B-format, a higher order Ambisonics (HOA) signal or mid-side stereo. For such audio signals, synthesis of the audio signal and parametric metadata is used to synthesize an audio scene to create a desired spatial perception.

[0108] Parametric metadata can be generated by different techniques. For example, Nokia's Spatial Audio Capture (OZO Audio) or Directional Audio Coding (DirAC) can be used. Both capture the sound field and represent it using parametric metadata. Parametric metadata may for example include: a directional parameter indicating the direction of each frequency band; a distance parameter indicating the distance of each frequency band; an energy splitting parameter indicating the ratio of the diffuseness to the total energy of each frequency band. For the direct version, each time-frequency bin can be considered as a sound source with an amplitude panning based on the directional parameter controlling the vector, while for the indirect (decoration-related) version it can be considered as a sound source with the energy splitting parameter controlling the differential gain.

[0109] The encoded audio content may be speech and / or music and / or general audio.

[0110] 3GPP IVAS (3GPP, Immersive Voice and Audio Services), currently under development, is expected to support new immersive voice and audio services such as mediated reality.

[0111] In some but not necessarily all examples, amplitude panning techniques may be used to create or locate sound objects. For example, the known method of vector-based amplitude panning (VBAP) may be used to locate sound sources.

[0112] A sound object may be repositioned by mixing a portion of the direct form of the object (attenuated and directionally filtered direct sound) with an indirect form of the object (eg, localized directional early reflections and / or diffuse reverberation).

[0113] Figure 2 An example of a system 100 for controlling presentation of spatial audio content 102 is illustrated.

[0114] In this example, the system 100 includes an apparatus 112 that receives the spatial audio content 102 and generates a first portion 113 (first audio content) of the spatial audio content 102 for a first device 130, and a second portion 111 (second audio content) of the spatial audio content 102 for a second device 120. In this example, the first device 130 is a handheld portable device. A handheld portable device is a device of a size and mass that allows it to be carried by hand. In some examples, its size and mass allow it to be moved relative to the user's body, for example, the user can easily hold it toward the user's ear with one hand. In some examples, its size and mass allow it to be carried in the user's pocket. In this example, the second device 120 is a spatial audio device, and the second portion 111 of the spatial audio content 102 is the spatial audio content-the spatial audio device 120 is configured to present the second portion 111 of the spatial audio content 102 as a presented sound scene 76 using a localized presentation sound source.

[0115] The spatial audio device 120 may be configured to present only spatial audio, or may be configured to present both spatial audio and the virtual visual scene 78 .

[0116] The stereo speakers are configured to present binaural audio content or other spatial audio content. The speakers transmit the sound source 140 (in Figure 2 The second portion 111 of the spatial audio content 102 is located in the sound space 60. The rendered sound scene 76 depends on the received second portion 111 of the spatial audio content 102 and, in some examples, on the personal rendering settings of the spatial audio device 120. The second portion 111 of the spatial audio content 120 has or is converted to have a format suitable for rendering by the second device 120.

[0117] The spatial audio device 120 may be a head-mounted audio output device, such as headphones. The headphones may use in-ear speakers, earmuffs with speakers, or near-ear speakers, for example, which are part of the "temples" for eyeglass frames.

[0118] The spatial audio device 120 may be a pass-through headset that allows the user to hear not only the audio rendered by the speaker or speakers of the spatial audio device 120 , but also clearly hear ambient audio, such as audio output from the handheld portable device 130 .

[0119] The spatial audio device 120 is configured to implement first-person perspective mediated reality. For example, the spatial audio device 120 may include a circuit system capable of tracking the user's head movement when the user wears the spatial audio device 120.

[0120] In some, but not necessarily all, examples, spatial audio device 120 may include a head-mounted display for one or both eyes of user 51 .

[0121] The handheld portable device 130 includes one or more speakers and is configured to cause presentation of the first portion 113 of the spatial audio content 102 via the one or more speakers.

[0122] Although in some examples the handheld portable device 130 is capable of combined playback from multiple speakers, it is not necessarily capable of spatial audio output. Therefore, the output audio 131 is heard as if it were emitted from the handheld portable device 130.

[0123] In contrast, spatial audio device 120 is capable of placing sound source 140 within sound space 60 .

[0124] In some, but not necessarily all, examples, handheld portable device 130 is a mobile cellular telephone.

[0125] The apparatus 112 decodes the received spatial audio content 102 and causes a first portion 113 of the spatial audio content 102 to be presented to the user 51 via the handheld portable device 130 and a second portion 111 of the spatial audio content 102 to be presented to the user 51 via the spatial audio device 120. The apparatus 112 includes a decoder 104 for decoding the spatial audio content 102. The decoding produces the spatial audio content 102 in a format that can be used to identify and individually process the sound sources 140. The decoded spatial audio content is provided to the rendering control block 106 and the renderer 110. The rendering control block 106 determines how the sound sources 140 within the spatial audio content 102 are to be presented. The determination is based on the received allocation information 108. The rendering control block 106 provides a control output to the renderer 110.

[0126] The renderer 110, under control of a control signal from the rendering control block 106, renders the spatial audio content received from the decoder 104 as a first part 113 for rendering by the handheld portable device 130 and as a second part 111 for rendering by the spatial audio device 120. The second part 111 is the spatial audio content.

[0127] The renderer 110 is configured to implement a first person perspective mediated reality with respect to the second portion 111 of the spatial audio content 102. Thus, the renderer 110 takes into account the viewpoint 54 of the user 51.

[0128] The second part 111 of the spatial audio content 102 may be transmitted to the spatial audio device 120 via any suitable link. In some but not necessarily all examples, the link is a wireless link, such as a radio link. The radio link may be provided via any suitable protocol such as Bluetooth or WLAN.

[0129] The first portion 113 of the spatial audio content 102 may be transmitted to the handheld portable device 130 via any suitable link. In some but not necessarily all examples, the link is a wireless link, such as a radio link. The radio link may be provided via any suitable protocol such as Bluetooth or WLAN.

[0130] In some, but not necessarily all, examples, the renderer 100 may include different renderers for the first portion 113 and the second portion 111 of the spatial audio content 102 .

[0131] The rendering control block 106 is configured to identify and control each sound source individually when required. It can control the renderer 110 to combine one or more sound sources within the rendered sound scene 76 in response to the second portion 111 of the spatial audio content 102 .

[0132] In some examples, the allocation information 108 is optional. For example, in an embedded codec, the lowest layer (monovoice) can be automatically transmitted as the first part 113 of the spatial audio content 102, and the other layers (spatial enhancement layers) can be provided as the second part 111 of the spatial audio content 102.

[0133] In other examples, the form of the rendered sound scene 76 may be controlled by the assignment information 108 .

[0134] In this example, the rendering control block 106 and the renderer 110 are housed within the same apparatus 112, in other examples, the rendering control block 106 and the renderer 110 may be housed in separate devices. In either implementation, an apparatus 112 is provided, including components for: simultaneously controlling content 131 rendered by a handheld portable device 130 and content 76 rendered by a spatial audio device 120; and in response to an action by the user 51, providing a rendering of a first portion 113 of the spatial audio content 102 to the user 51 via the handheld portable device 130 instead of the spatial audio device 120, instead of the second portion 111.

[0135] The apparatus 112 may include means, such as the presentation control block 106 , for determining a first portion 113 of the spatial audio content 102 and a second portion 111 of the spatial audio content 102 in response to a user action.

[0136] The apparatus 112 also includes means for causing, in response to an action by the user, simultaneous presentation to the user of a first portion 113 of the spatial audio content 102 via the handheld portable device 130 and presentation of a second portion 111 of the spatial audio content 102 via the spatial audio device 120. The spatial audio device 120 presents the second portion 111 instead of the first portion 113, and the handheld portable device 130 presents the first portion 113 instead of the second portion 111. In some examples, the handheld portable device 130 presents only the first portion 113, and the spatial audio device 120 presents only the second portion 111.

[0137] In some, but not necessarily all, examples, spatial audio content 102 is received by device 112 from a communication network. Spatial audio content 102 may, for example, define a sound scene, or may, for example, be an immersive call. The immersive call is provided in a downlink communication link from a remote person, and in this example, but not necessarily all, a first portion 113 of the spatial audio content 102 may include the remote person's voice and / or ambient sounds.

[0138] The presentation control module 106 can be configured to control whether the sound source 140 is presented as a plot content or as a non-plot content. The plot content has a position in the sound space 60 corresponding to a position in the real space 50. The location of the sound source relative to the user 51 is controlled by the first-person perspective mediated reality. In contrast, the non-plot content has no position in the real space 50 and is not modified due to the first-person perspective mediated reality. Examples of non-plot content include, for example, ambient sounds or a "narrator" voice.

[0139] Whether the sound source 140 is presented as diegetic content or non-diegetic content may depend on whether the sound source 140 is presented on the handheld portable device 130 or the spatial audio device 120. For example, the first portion 113 of the spatial audio content 102 may be a non-diegetic sound source 140. The second portion 110 of the spatial audio content 102 may be diegetic content and / or non-diegetic content.

[0140] exist Figure 3A , Figure 3B and Figure 3C In the example of FIG. 5 , the user action that causes the first portion 113 of the spatial audio content 102 to be presented to the user via the handheld portable device 130 and the second portion 111 of the spatial audio content 102 to be presented to the user via the spatial audio device 120 is raising the handheld portable device 130 toward the ear of the user 51. Of course, it should be appreciated that different user actions (including, for example, interactions and / or indications) may also cause such presentation.

[0141] Figure 3AAn example is illustrated in which a handheld portable device 130 presents 131 a first portion 113 of a spatial audio content 102 .

[0142] Figure 3B An example is illustrated in which the spatial audio device 120 renders the second portion 111 of the spatial audio content 102 as a rendered sound scene 76 including sound sources 140. Rendering the sound scene 76 includes rendering one or more sound sources at respective locations in the sound space 60.

[0143] Figure 3C The illustration shows that in response to an action of the user 51, the first portion 113 of the spatial audio content 102 is simultaneously presented to the user 51 via the handheld portable device 130, and the second portion 111 of the spatial audio content 102 is presented via the spatial audio device 120. In this example, the action performed by the user to cause the first portion 113 of the spatial audio content 102 to be presented to the user 51 via the handheld portable device 130 is raising the handheld portable device 130 toward the user's left ear 51.

[0144] In this example, the apparatus 112 includes means for communicating with the handheld portable device 130 and / or the spatial audio device 120 to determine when the user's action occurs. For example, this may be detected by using a proximity detector for the handheld portable device 130 that detects the proximity of the handheld portable device 130 to the ear of the user 51 by, for example, measuring acoustic impedance, or by detecting the proximity of the handheld portable device 130 to the spatial audio device 120. Alternatively, the handheld portable device 130 may use an inertial measurement unit to measure the movement of the handheld portable device 130 and detect movement corresponding to the upward lifting of the mobile device 130 in combination with the upright orientation of the handheld portable device 130.

[0145] In other examples, spatial audio device 120 may include a proximity detector for detecting the proximity of handheld portable device 130 .

[0146] Using the lifting of the handheld portable device 130 to the ear of the user 51 to cause the spatial audio data 102 to be divided into a first portion 113 for the handheld portable device 130 and a second portion 111 for the spatial audio device 120 is particularly useful when the spatial audio content 102 is related to an incoming immersive phone call. An immersive phone call is a phone call that includes spatial audio content.

[0147] In the case where the incoming spatial audio content 102 is an immersive phone call, the first portion 113 of the spatial audio content 102 may be content presenting the voice of the caller. The caller's voice may then be removed from the spatial sound scene defined by the spatial audio content of the incoming immersive phone call to produce the second portion 111 of the spatial audio content 102 presented by the spatial audio device 120 as the presented sound scene 76.

[0148] In some examples, the spatial audio content 102 may be rendered entirely by the spatial audio device 120 before the user action causes the spatial audio content 102 to be divided into the first portion 113 for the handheld portable device 130 and the second portion 111 for the spatial audio device 120. Thus, prior to the user's action, the first portion 113 and the second portion 111 of the spatial audio content 102 may be rendered simultaneously by the spatial audio device 120.

[0149] The following is an example of a use case of the system 100. A user 51 is wearing a spatial audio device 120 and a handheld portable device 130 is in his pocket. A friend of the user calls him, for example, using IVAS, via an immersive phone call. The user 51 is alerted to the incoming immersive call via the ringtone of the handheld portable device 130, via the content presented by the spatial audio device 120, or by both. The user 51 wants to receive the incoming immersive call on his handheld portable device 130, but also wants to hear the ambient signal of his friend. The user 51 answers the incoming immersive call by touching the handheld portable device 130 against the spatial audio device 120 at his ear. The presentation control module 106, which may be part of the call software of the handheld portable device 130, assigns the caller's voice (the friend's voice) to the handheld portable device 130 as a first portion 113 of the spatial audio content 102 and assigns the ambient signal to the spatial audio device 120 as a second portion 111 of the spatial audio content 102. The spatial audio device 120 allows the user to not only hear the audio presented by one or more speakers of the spatial audio device 120, but also clearly hear the ambient audio such as the audio output from the handheld portable device 130. The user hears the ambient signals emitted around him as the rendered sound scene 76 presented by the spatial audio device 120, and uses the handheld portable device 130 in a conventional manner to hear the voice of the caller. The user 51 can, for example, switch the handheld portable device 130 from ear to ear, and in some examples, can be able to use the handheld portable device 130 in a hands-free mode. The user interface on the handheld portable device 130 and / or the spatial audio device 120 enables the user 51 to control the assignment information 108. For example, the assignment information 108 can be changed to control the volume of the second portion 111 of the spatial audio content presented by the spatial audio device 120 and / or to control the volume of the first portion 113 of the spatial audio content 102 presented by the handheld portable device 130.

[0150] exist FIG. 4A to FIG. 4C as well as FIG. 5A to FIG. 5CIn the illustrated example, the handheld portable device 130 is used to present the first part 113 of the spatial audio content 102, and the spatial audio device 120 is used to present the second part 111 of the spatial audio content 102. Therefore, the handheld portable device 130 and the spatial audio device 120 are used simultaneously for the presentation of audio content. Therefore, the audio content presented from one device can affect the user's perception of the audio content presented by the other device. In the illustrated example, the second part 111 of the spatial audio content is suitable for moving one or more sound sources 140 within the second part 111 of the spatial audio content to avoid this problem. The sound source 140 that is moved is an additional sound source. For example, it can be an internal diegetic sound source from within the immersive call content, rather than the main voice of the caller, or it can be an external additional sound source from the content, rather than the immersive call.

[0151] The internal additional sound source may be, for example, another person speaking (not the surrounding background noise). This may occur, for example, if the immersive call is a multi-party conference call. In some examples, when the user of the handheld portable device 130 and the spatial audio device 120 is conducting a multi-party conference call with a second remote party, a third remote party, a fourth remote party, etc., then the user may select which remote party's voice is presented from the handheld portable device 130, while the other two remote parties will continue to be presented in the spatial audio output of the spatial audio device 120. The user may individually select which of the multiple remote parties participating in the multi-party conference call to separate to the handheld portable device 130 and the spatial audio device 12, respectively.

[0152] The external additional sound source may, for example, be from ongoing content that is not stopped or paused and continues (at least partially) when the immersive call is received, or may be a new content separate from the immersive call that starts after the immersive call is received.

[0153] In some, but not necessarily all, examples, the user may select the additional sound sources using, for example, a user interface of handheld portable device 130 or spatial audio device 120 .

[0154] Examples of external additional sound sources include, for example, music playback, messages, advertisements, and contextual guidance.

[0155] exist FIG. 4A to FIG. 4C as well as FIG. 5A to FIG. 5C In FIG. 1 , the spatial audio scene 76 defined by the second portion 111 of the spatial audio content 102 is modified depending on the position 133 of the handheld portable device 130 .

[0156] For example, the location of the additional source 140' is moved 144 based on the user action 137, such as movement of the handheld portable device 130 to a new location.

[0157] Figure 4A The location of a specific sound source 140 ′ in the spatial audio content 102 before the spatial audio content 102 is divided into a first portion 113 for the handheld portable device 130 and a second portion 111 for the spatial audio device 120 in response to the user action 137 is illustrated.

[0158] Figure 4B The situation is illustrated after the user 51 has performed an action 137 that causes the spatial audio content 102 to be divided into a first part 113 and a second part 111. The first part 113 is or will be presented by the handheld portable device 130 as audio content 131 in the left ear of the user in which the handheld portable device 130 is located. Figure 4C , the rendering control module 106 causes the renderer 110 to adapt the second portion 111 of the spatial audio content 102 to change the position of the spatial audio source 140 ′. In this example, the sound source 140 ′ is moved 144 to be closer to the right ear of the user 51 .

[0159] Figure 5A , Figure 5B and Figure 5C Similar to Figure 4A , Figure 4B and Figure 4C However, there are now two original sound sources A, B, and the initial position of the specific sound source B 140 that is moved is different. Figure 5B In the example of , a particular sound source 140 ′ of the rendered sound scene 76 is directly adjacent to the user's left ear and is moved 144 away from the user's left ear to create a spatial separation between the additional sound source 140 ′ and the handheld portable device 130 .

[0160] The allocation information 108 may provide information identifying the location of the handheld portable device 130 in three-dimensional space. For example, this may indicate that it is near the user's left or right ear, or may provide a three-dimensional position in real space 50.

[0161] The presentation control module 106 is configured to determine the distance separation between the position of the handheld portable device 130 and the additional sound source 140'. If the separation distance is below a threshold, the presentation control module 106 determines that the additional sound source 140' is suboptimally placed. For example, it may determine that Figure 4B There is a spatial conflict as shown in Figure 5B The existence illustrated in is at the same location.

[0162] The rendering control module 106 modifies the spatial audio sound scene 76 by moving at least the additional sound source 140'. In some but not necessarily all examples, only the additional sound source 140' determined to be suboptimally placed is moved. In other examples, the entire rendering sound scene 76 is rotated relative to the user 51 to remove spatial conflicts or be in the same location.

[0163] exist Figure 4A and Figure 5A In the example of , the spatial audio content 102 is presented to the user 51 using a first person perspective mediated reality before the user action 137 causes a modification of the spatial audio content 102 by dividing the spatial audio content 102 into a first portion 113 for the handheld portable device 130 and a second portion 111 for the spatial audio device 120. For example, when the user 51 changes their viewpoint 54, the additional sound source 140' may have a fixed position in the real space.

[0164] exist Figure 4C and Figure 5C In the example of FIG. 5 , after a user action 137 causes a modification of the spatial audio content 102 by dividing the spatial audio content 102 into a first portion 113 for a handheld portable device 130 and a second portion 111 for a spatial audio device 120, the spatial audio content 102 including a repositioned additional sound source 140′ is presented to the user 51 using a first-person perspective mediated reality. For example, the repositioned additional sound source 140′ may have a fixed position in real space when the user 51 changes their viewpoint 54.

[0165] In some, but not necessarily all, examples, the first portion 113 of the spatial audio content 102 relates to a first speech associated with a first ear of the user 51, where the handheld portable device 130 is positioned at the first ear. The second portion 111 of the spatial audio content relates to at least a second, different speech, where the spatial audio scene 76 defined by the second portion 111 of the spatial audio content 102 is modified to place the second speech 140′ at a second ear of the user, opposite the first ear.

[0166] In some but not necessarily all examples, the first voice and the second different voice may relate to different sound sources within the immersive phone call. In such an example, FIG. 4A to FIG. 4C and FIG. 5A to FIG. 5C An example of this may occur in pre-processing prior to any rendering of the spatial audio content 102. The spatial audio content 102 of an immersive phone call may only be rendered in Figure 4C and Figure 5C The modification of the second portion 111 of the spatial audio content 102 illustrated in is presented for the first time only afterwards.

[0167] FIG. 6A to FIG. 6C An example is illustrated in which the user 51 edits the second portion 111 of the spatial audio content 102. In this example, the second portion 111 of the spatial audio content 102 is edited in response to movement of the handheld portable device 130.

[0168] In some, but not necessarily all, examples, the user 51 may control the allocation of the sound sources 140 between the first portion 113 for the handheld portable device 130 and the second portion 111 for the spatial audio device 120 . This may be accomplished by modifying the allocation information 108 .

[0169] In some but not necessarily all examples, user 51 may control the assignment of sound source 140 by moving sound source 140 from being presented by spatial audio device 120 as part of presentation scene 76 to being presented by handheld portable device 130, and / or control the reallocation of sound source 140 from being presented by handheld portable device 130 to being presented by spatial audio device 120.

[0170] In other examples, a user may control the balance between the audio output by the handheld portable device 130 and the spatial audio device 120 , for example by controlling the volume of the devices individually or by muting the sound sources 140 presented by either device.

[0171] exist Fig. 6A In the example of , a main sound source Y is presented 131 by a handheld portable device 130 close to the left ear of the user 51. At the same time, the spatial audio device 120 presents an additional sound source 140' as the presented sound scene 76. The additional sound source 140' is an auxiliary sound source X. The main sound source Y can be, for example, speech in an immersive phone call. The additional sound source 140' can be, for example, different speech or other different content in the immersive call.

[0172] like Figure 6B , the system 100 is configured to enable the user 51 to control the allocation of sound sources 140 between the rendering devices 120, 130. In this example, the user uses a user interface on the handheld portable device 130 to switch between a primary sound source Y and a secondary sound source X. The primary sound source Y is now rendered as the sound source 140 in the rendering sound scene 76 rendered by the spatial audio device 120. This is achieved by modifying the second portion 111 of the spatial audio content 102 in response to a control signal from the rendering control module 106. The secondary sound source X is now rendered by the handheld portable device 130. This is achieved by modifying the first portion 113 of the spatial audio content 102 under the control of the rendering control module 106.

[0173] like Figure 6C, the user can now lower the handheld portable device 130 so that the user can no longer hear the auxiliary sound object X. The user can then listen to the primary sound object Y as a sound source 140 in the rendered sound scene 76 rendered by the spatial audio device 120.

[0174] In some, but not necessarily all, examples, the action 135 of lowering the handheld portable device 120 may cause the secondary sound source X to be removed from the second portion 111 of the spatial audio content 102. Subsequent raising of the handheld portable device to the user's ear may then cause the primary sound source Y to be delivered back to the first portion 113 of the spatial audio content 102. As a result, the user has returned to a state similar to Fig. 6A In the situation illustrated in , except that the additional sound source 140 ′ has been removed from the sound scene 76 rendered by the spatial audio device 120 .

[0175] exist Figure 6C In the example of , if the user 51 is to speak during the immersive call, the spatial audio device 120 will need to have a microphone, or an additional microphone 139 will need to be provided separately, such as a lavalier microphone, or provided by the handheld portable device 130.

[0176] Fig. 7A An example is illustrated in which a spatial audio device 120 is presenting spatial audio content 102 to a user 51. The presented spatial audio content defines a sound scene 76 comprising a plurality of sound sources 140. Figure 7B and Figure 7C It is illustrated that the user can perform an action that causes the sound source 140 to be removed from the rendered sound scene 76 so that it is rendered from the handheld portable device 130 instead. Figure 2 , which corresponds to a change in the allocation information 108, so that the rendering control block 106 controls the renderer 110 to place the specific sound source 140 into the first part 113 of the spatial audio content rendered by the handheld portable device 130. The remaining spatial audio content minus the specific sound source 140 is provided to the spatial audio device 120 as the second part 111 of the spatial audio content 102 to be rendered as the sound scene 76.

[0177] Therefore, the apparatus 112 includes components for simultaneously controlling the content presented by the handheld portable device 130 and the content presented by the spatial audio device 120, and in response to an action of the user, providing the user with a first portion 113 of the spatial audio content 102 presented via the handheld portable device 130 instead of a second portion 111 of the spatial audio content 102 presented via the spatial audio device 120.

[0178] In this example, upon the user's action ( Fig. 7A ), there is simultaneous presentation. The device 112 includes means for enabling selection of a first portion 113 of the spatial audio content 102 via the user 51 using a first person perspective mediated reality. Figure 7C is exemplified in . exist Figure 7C In an example, the user may select 79 the desired sound source 140 by changing the viewpoint 74 of the user 51 used to select the desired sound source 140 .

[0179] The selected first portion 113 of the spatial audio content 102 may be modified prior to presentation by the handheld portable device 130 , for example to emphasize a particular audio source in a mono downmix.

[0180] In this example, the user action is raising the handheld portable device 130 to the user's head. As previously described, the apparatus 112 simultaneously presents the first portion 113 of the spatial audio content 102 via the handheld portable device 130 and the second portion 111 of the spatial audio content 102 via the spatial audio device 120 to the user 51 .

[0181] In some, but not necessarily all, examples, the first portion 113 includes a single sound source 140 selected by the user 51. In other examples, the first portion 113 includes multiple sound sources 140 selected by the user 51 separately and sequentially.

[0182] Figure 8 An example of a method 200 that may be performed by an apparatus 112 is illustrated. At block 202, the apparatus 112 receives an immersive call. At block 204, the immersive audio 102 is rendered by the spatial audio device 120. At block 206, the apparatus 112 detects that the user 51 has raised the handheld portable device 130 close to the user's ear. At block 208, in response to the rendering control block 106, a first portion 113 of the spatial audio content 102 is created by the renderer 110. In some examples, all of the spatial audio content 102 of the immersive call is downmixed to form a single first portion 113 rendered by the handheld portable device 130. In this example, the renderer 110 may not provide any second portion 111 of the spatial audio content 102 to the spatial audio device 120. In other examples, the renderer 110 controls the second portion 111 of the spatial audio content under the control of the rendering control block 106 to include one or more specific sound sources 140. In this example, the second portion 111 of the spatial audio content 102 includes the portion of the spatial audio content 102 that remains after subtracting the first portion 113 .

[0183] At block 210, the apparatus 112 responds to a change in the viewpoint 54 of the user 51. This change in viewpoint is provided as the assignment information 108 and causes a change in the selection of audio content presented by the handheld portable device 130.

[0184] For example, a mono downmix created from the spatial audio content 102 may be created with an emphasis on one or more sound sources in a particular direction of arrival corresponding to the viewpoint 54 of the user 51. Likewise, the selection of the particular sound source 140 may correspond to a sound source aligned with the viewpoint 54 of the user 51. Thus, as the viewpoint 54 of the user 51 changes, the content of the first portion 113 of the spatial audio content 102 also changes.

[0185] At block 212 , the first portion 113 of the spatial audio content 102 is delivered to the handheld portable device 130 for presentation.

[0186] 9A to 9E Detailed illustration of how the apparatus 112 enables a user to select a first portion of the spatial audio content using a first-person perspective mediated reality. Fig.9A A virtual user 71 is illustrated, which has a viewpoint 74 within the sound space 60 including the sound source 140. As previously described, in the first-person perspective mediated reality, there is a correspondence between the virtual user 71 and the user 51, and between the viewpoint 74 of the virtual user 71 and the viewpoint 54 of the user 51. The user 51 changes the viewpoint 54 by changing the orientation of the user's head. This changes the virtual viewpoint 74.

[0187] In this immersive audio scene, the rendered sound scene 76 includes a plurality of sound sources 140. In this example, these sound sources are fixed in space so that when the user 51 changes his viewpoint 54, the position of the sound sources 140 relative to the user 51 also changes.

[0188] exist Fig. 9B In the example of , the user has raised the handheld portable device 130 to his left ear or has performed any other suitable gesture to cause the first part 113 of the spatial audio content 102 to be presented via the handheld portable device 130 and the second part 111 of the spatial audio content 102 to be presented via the spatial audio device 120. The first part 113 of the spatial audio content emphasizes the sound source 140 that is aligned with or closest to the viewpoint 74 of the virtual user 71. In the example of mono down-conversion, the mono down-conversion is emphasized in the direction of the virtual viewpoint 74. In the example of a sound object or the selection of a specific sound source 140, the viewpoint 74 selects the sound source 140 that is aligned with or closest to the virtual viewpoint 74. The selected sound source 140 is included in the first part 113 of the spatial audio content 102, but is not included in the second part 111 of the spatial audio content 102. Fig. 9BIn the example of FIG. 1 , the sound source 140 labeled “2” is selected and emphasized in the first portion 113 of the spatial audio content 102 .

[0189] exist Fig. 9C In the example of , the user 51 has changed his viewpoint 54 by changing the orientation of his head. This causes the viewpoint 74 of the virtual user 71 to change accordingly. The change in the viewpoint 74 of the virtual user changes which part of the sound scene is emphasized in the first part 113 of the spatial audio content 102. Fig. 9B As described, the only difference is the change of viewpoint 74. Fig. 9C In the example of FIG. 4 , the sound source 140 labeled as “4” is selected and emphasized in the first portion 113 of the spatial audio content 102 .

[0190] exist Fig.9D In the example of , the user 51 has removed the spatial audio device 120. The selected sound source 140 is now frozen. Fig.9D In the example of , the sound source 140 labeled "4" is selected and emphasized. The subsequent movement of the user's head and the change of the viewpoint 54 of the user 51 do not change the viewpoint 74 of the virtual user 71. Therefore, the selection of the emphasized sound source 140 does not change, as shown in FIG. Fig.9E exemplified in . exist Fig.9E In the example of , the sound source labeled “4” remains the sound source 140 emphasized in the first portion 113 of the spatial audio content 102 .

[0191] Therefore, it will be appreciated that, although the user is wearing a spatial audio device 120 with head tracking capabilities, the first-person perspective mediated reality is enabled. The viewpoint 54 of the user 51 and the virtual viewpoint 74 of the virtual user 71 correspond to and track each other. The virtual viewpoint 71 is used to emphasize a specific sound source 140 aligned with the virtual viewpoint 74 within the spatial audio content. The specific sound source 140 is emphasized in the first portion 113 of the spatial audio content 102. The emphasized sound source 140 can be changed by changing the user's viewpoint 54, which changes the viewpoint 74 of the virtual user. However, once the spatial audio device 120 is removed, the head movement of the user 51 can no longer be tracked and the user's viewpoint 54 is decoupled from the virtual viewpoint 74 of the virtual user. Therefore, changes in the viewpoint 54 of the user 51 will not cause changes in the virtual viewpoint 74 of the virtual user 71, nor will they cause changes in the selected sound source 140 emphasized in the first portion 113 of the spatial audio content 102.

[0192] exist Fig. 10A and Fig. 10BIn the example of , the user 51 selects a particular sound source 140, and that sound source is emphasized, regardless of a subsequent relative change in alignment between the virtual viewpoint 74 and the selected sound source 140. This lack of alignment may occur, for example, because the selected sound source 140 is moving and / or because the user 51 changes his viewpoint 54. As a result of this, the selected moving sound source 140 is presented as if the moving sound source were stationary.

[0193] exist Fig. 10A , the viewpoint 74 of the virtual user 71 is aligned with a specific sound object 140. This alignment selects the specific sound object 140, which is then included in the first portion 113 of the spatial audio content and presented by the handheld portable device 130. The selection of the sound object 140 may be a result of an action of the user 51.

[0194] exist Fig. 10B , the selected sound source 140 has moved, but the viewpoint 74 of the virtual user 71 has not moved. Although the viewpoint 74 of the virtual user 71 is no longer directed at the selected sound object 140, the selected sound object 140 remains selected and included in the first portion 113 of the spatial audio content presented by the handheld portable device 130. The selected sound object 140 remains selected and included in the first portion 113 of the spatial audio content presented by the handheld portable device 130 not based on the user's viewpoint but based on tracking. The position of the selected sound object 140 can be tracked and therefore, the manner in which it is included in the first portion 113 of the spatial audio content presented by the handheld portable device 130 can change as the tracked position of the selected sound object 140 changes. Therefore, in Fig. 10A and Fig. 10B In FIG. 1 , sound objects “1”, “3”, and “4” are rendered by the spatial audio device 120 and controlled by the second portion 111 of the spatial audio content 102. Fig. 10A and Fig. 10B , the sound object “ 2 ” is rendered by the handheld portable device 130 under control of the first portion 113 of the spatial audio content 102 .

[0195] In the example of 10C, the spatial audio device 120 has been removed. Nevertheless, the selected spatial sound object 2 remains selected and continues to be rendered by the handheld portable device 130 under control of the first portion 113 of the spatial audio content 102 - even though it is moving.

[0196] Fig. 10D1. It illustrates that even though the user 51 changes his viewpoint 54, the selected sound object "2" continues to be rendered by the handheld portable device 130 under control of the first portion 113 of the spatial audio content 102, as previously described in FIG. 10A to FIG. 10C middle.

[0197] Return to see Fig. 9B and Fig. 9C , it will be appreciated that when the user 51 changes his viewpoint 54, he changes the virtual user's viewpoint 74, which changes the sound source 140 emphasized in the first part 113 of the spatial audio content 102 presented by the handheld portable device 130. In the example where the emphasized sound source 140 is a sound object, the remaining sound objects of the sound scene are presented by the spatial audio device 120. That is, the first part 113 of the spatial audio content 102 includes sound object "2" and the second part 111 of spatial audio objects includes sound objects "1", "3", "4". When the user changes orientation, changes the viewpoint 54, then the user's virtual viewpoint 74 also changes. This is in the example where the emphasized sound source 140 is a sound object. Fig. 9C As previously described, this will change the selected sound object. The first portion 113 of the spatial audio content 102 includes sound object "4". Therefore, the content of the second portion 111 of the spatial audio content now includes sound objects "1", "2", "3". In addition, since the spatial audio device 120 enables first-person perspective mediated reality, as a result of the virtual user's viewpoint changing, Fig. 9B and Fig. 9C By comparison, the positions of the sound objects "1" and "3" presented to the user via the spatial audio device 120 have been rotated relative to the user. Similar considerations apply to FIG. 10A to FIG. 10D .

[0198] Thus, it will be appreciated that, after determining a user action of the first portion 113 of the audio content 102, the apparatus 112 enables the use of a first-person perspective mediated reality to adapt the second portion 111 of the spatial audio content 102 while using at least the spatial audio device 120. The apparatus 112 also enables presentation of the adapted second portion 111 of the spatial audio content 102 via at least the spatial audio device 120.

[0199] In some but not necessarily all examples, the user 51 may be able to lock on to the selected object without locking on to the spatial audio scene. As a result, the apparatus 112 is able to present the second portion 111 of the spatial audio content 102 via the spatial audio device 120, wherein the second portion 111 has been adapted according to the first-person perspective mediated reality.

[0200] The user is also able to lock the rendered sound scene 76 in response to a subsequent user action. The apparatus 112 is then configured to, after the subsequent user action, disable the adaptation of the second portion 111 of the spatial audio content according to the first-person perspective mediated reality and cause the second portion 111 of the spatial audio content 102 to be rendered according to the fixed first-person perspective via the spatial audio device 120.

[0201] exist Fig. 9B In the example of FIG. 1 , the removal of the spatial audio device 120 simultaneously locks the object and locks the scene.

[0202] Fig.11A Picture shows Figure 2 An example of an implementation of the system 100 illustrated in FIG. 1 , the operation of which has been described in the previous description. In this example, the allocation information 108 is provided from the handheld portable device 130 to a remote server 180 including the rendering control block 106. In this example, the server 180 also includes the renderer 110. However, in other examples, it can be located in the handheld portable device 130.

[0203] The renderer 110 provides the first portion 111 and the second portion 111 of the spatial audio content 102 to the handheld portable device 130 under the control of the rendering control block 106 in a manner that allows the handheld portable device 130 to render the first portion 113, and provides the second portion 111 to the spatial audio device 120 for rendering. The handheld portable device 130 provides the allocation information 108 to the server 180. This may include, for example, an indication of a user action associated with a control command.

[0204] Fig. 11B Picture shows Figure 2 1, the operation of which has been described in the previous description. In this example, the allocation information 108 is provided internally within the handheld portable device 130 including the rendering control block 106. In this example, the handheld portable device 130 includes the renderer 110. The server 180 provides the spatial audio content 102.

[0205] The renderer 110 provides the first portion 111 and the second portion 111 of the spatial audio content 102 to the handheld portable device 130 and provides the second portion 111 to the spatial audio device 120 for rendering in a manner that allows the handheld portable device 130 to render the first portion 113 under the control of the rendering control block 106. The assignment information 108 may, for example, include an indication of a user action associated with the control command.

[0206] Fig. 12AAn example of a controller 400 is illustrated. Such a controller may be used to control the performance of any or all functions of apparatus 112, control the performance of any or all functions of handheld portable device 130, control the performance of any or all functions of spatial audio device 120.

[0207] Controller 400 may be implemented as controller circuitry.Controller 400 may be implemented solely in hardware, may have certain aspects in software including firmware alone, or may be a combination of hardware and software (including firmware).

[0208] like Fig. 12A As illustrated in the figure, the controller 400 can be implemented using instructions that enable hardware functionality, such as by using executable instructions of a computer program 412 in a general or special purpose processor 402, which can be stored in a computer-readable storage medium (disk, memory, etc.) executed by such a processor 402.

[0209] The processor 402 is configured to read from and write to the memory 410. The processor 402 may also include an output interface via which the processor 402 outputs data and / or commands, and an input interface via which data and / or commands are input to the processor 402.

[0210] The memory 410 stores a computer program 412, which includes computer program instructions (computer program code) that control the operation of the device 112 when loaded into the processor 402. The computer program instructions of the computer program 412 provide logic and routines that enable the device to perform the methods illustrated in Figures 1 to 13. By reading the memory 410, the processor 402 can load and execute the computer program 412.

[0211] Thus, the device 112 comprises:

[0212] at least one processor 402; and

[0213] at least one memory 410 comprising computer program code,

[0214] The at least one memory 410 and the computer program code are configured to, together with the at least one processor 402, cause the apparatus 112 to at least perform:

[0215] Simultaneously controlling content presented by handheld portable device 130 and content presented by spatial audio device 120; and

[0216] In response to the user's action, presentation of the first portion 113 of the spatial audio content 102 instead of the second portion is provided to the user via the handheld portable device instead of the spatial audio device 120 .

[0217] Thus, the device 112 may include:

[0218] at least one processor 402; and

[0219] at least one memory 410 comprising computer program code,

[0220] The at least one memory 410 and the computer program code are configured to, together with the at least one processor 402, cause the apparatus 112 to at least perform:

[0221] Simultaneously controlling content presented by handheld portable device 130 and content presented by spatial audio device 120; and

[0222] In response to the user's action, presentation of the first portion 113 , rather than the second portion, of the spatial audio content 102 is provided to the user 51 via the handheld portable device rather than the spatial audio device 120 .

[0223] like Fig. 12B , the computer program 412 may arrive at the device 112 via any suitable delivery mechanism 420. The delivery mechanism 420 may be, for example, a machine-readable medium, a computer-readable medium, a non-transitory computer-readable storage medium, a computer program product, a memory device, a recording medium such as a compact disk read-only memory (CD-ROM) or a digital versatile disk (DVD) or a solid-state memory, an article of manufacture including or tangibly embodying the computer program 412. The delivery mechanism may be a signal configured to reliably transmit the computer program 412. The device 112 may propagate or transmit the computer program 412 as a computer data signal.

[0224] In at least some examples, the computer program instructions are configured to cause the apparatus to at least perform the following operations:

[0225] Simultaneously controlling content presented by handheld portable device 130 and content presented by spatial audio device 120; and

[0226] In response to the user's action, the first portion 113 of the spatial audio content 102 , rather than the second portion, is caused to be presented to the user 51 via the handheld portable device rather than the spatial audio device 120 .

[0227] The computer program instructions may be included in a computer program, a non-transitory computer readable medium, a computer program product, a machine readable medium. In some but not necessarily all examples, the computer program instructions may be distributed over more than one computer program.

[0228] Although memory 410 is illustrated as a single component / circuitry, it may be implemented as one or more separate components / circuitry, some or all of which may be integrated / removable and / or may provide permanent / semi-permanent / dynamic / cached storage.

[0229] Although processor 402 is illustrated as a single component / circuitry, it may be implemented as one or more separate components / circuitry, some or all of which may be integrated / removable.Processor 402 may be a single-core or multi-core processor.

[0230] References to "computer-readable storage medium," "computer program product," "tangibly embodied computer program," etc. or "controller," "computer," "processor," etc. should be understood to include not only computers having different architectures, such as single / multi-processor architectures and sequential (von Neumann) / parallel architectures, but also special purpose circuits, such as field programmable gate arrays (FPGAs), application specific circuits (ASICs), signal processing devices, and other processing circuit systems. References to computer programs, instructions, code, etc. should be understood to cover software for a programmable processor or firmware, such as, for example, the programmable content of a hardware device - whether instructions for a processor, or configuration settings for a fixed function device, gate array or programmable logic device, etc.

[0231] In this application, the term "circuitry" may refer to one or more or all of the following:

[0232] (a) pure hardware circuit implementation (e.g. implementation in analog and / or digital circuits only) and

[0233] (b) a combination of hardware circuitry and software such as (where applicable):

[0234] (i) a combination of (one or more) analog and / or digital hardware circuits and software / firmware, and

[0235] (ii) any portion of hardware processor(s) with software (including digital signal processor(s), software and memory(s) that work together to enable a device such as a mobile phone or server to perform various functions), and

[0236] (c) Hardware circuit(s) and / or processor(s), such as microprocessor(s) or portion(s) of microprocessor(s), that require software (e.g., firmware) to operate, but that software may not be present when not required to operate.

[0237] This definition of circuitry applies to all uses of the term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of only a portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. By way of example and where applicable to a particular claim element, the term circuitry also covers a baseband integrated circuit for a handheld portable device, or a similar integrated circuit in a server, cellular network device, or other computing or network device.

[0238] The blocks illustrated in Figures 1 to 13 may represent steps in a method and / or code segment in a computer program 412. The illustration of a particular order of blocks does not necessarily mean that the blocks have a required or preferred order and the order and arrangement of the blocks may be changed. In addition, it is possible to omit some blocks.

[0239] Where a structural feature has been described, it may be replaced by means of a component for performing one or more of the functions of the structural feature, whether that function or functions is explicitly or implicitly described.

[0240] The examples described above use the app as an enabling component for:

[0241] Automotive systems; telecommunications systems; electronic systems, including consumer electronics; distributed computing systems; media systems for generating or presenting media content, including audio, visual and audiovisual content, and mixed, mediated, virtual and / or augmented reality; personal systems, including personal health systems or personal fitness systems; navigation systems; user interfaces, also known as human-machine interfaces; networks, including cellular, non-cellular and optical networks; ad hoc networks; the Internet; the Internet of Things; virtual networks; and related software and services.

[0242] The term "comprising" is used in this document in an inclusive rather than exclusive sense. That is, any reference to X that includes Y indicates that X may include only one Y or may include more than one Y. If the exclusive meaning of "comprising" is intended, it is made clear in the context by referring to "including only one .." or using "consisting of . . . ".

[0243] In this description, reference has been made to various examples. The description of features or functions associated with an example indicates that those features or functions are present in the example. Whether explicitly stated or not, the use of the term "example" or "for example" or "may" or "might" in this article indicates that such features or functions are present in at least the example described, whether or not described as an example, and they may but not necessarily be present in some or all other examples. Therefore, "example", "for example", "may" or "might" refers to a specific instance in a class of examples. The attributes of an instance may be attributes of only that instance, or attributes of the class, or attributes of a subclass of the class, which includes some but not all instances in the class. Therefore, features described with reference to one example rather than with reference to another example are implicitly disclosed, and may be used as part of a working combination in the other example where possible, but do not necessarily have to be used in the other example.

[0244] Although the embodiments have been described in the preceding paragraphs with reference to various examples, it should be appreciated that modifications to the examples given can be made without departing from the scope of the claims.

[0245] Features described in the preceding description may be used in combinations other than those explicitly described above.

[0246] Although functions have been described with reference to certain features, those functions may be performable by other features whether described or not.

[0247] Although features have been described with reference to certain embodiments, those features may also be present in other embodiments whether described or not.

[0248] The terms "a", "an" or "the" used in this document have an inclusive, rather than exclusive, meaning. That is, unless the context clearly indicates otherwise, any reference to X that includes one / the indicates that X may include only one Y or may include more than one Y. If the exclusive meaning of "a" or "the" is intended, the context will make it clear. In some cases, the use of "at least one" or "one or more" is used to emphasize an inclusive meaning, but the absence of these terms should not be used to infer and exclusive meaning.

[0249] The presence of a feature (or combination of features) in a claim is a reference to the feature (or combination of features) itself, and also a reference to features that achieve substantially the same technical effect (equivalent features). Equivalent features include, for example, features that are variants and achieve substantially the same result in substantially the same way. Equivalent features include, for example, features that perform substantially the same function in substantially the same way to achieve substantially the same result.

[0250] In this description, reference has been made to various examples using adjectives or adjective phrases to describe characteristics of examples. Such description of characteristics associated with examples indicates that the characteristics exist in some examples exactly as described, and in other examples substantially as described.

[0251] Whether explicitly stated or not, the use of the term "example" or "for example" or "may" or "might" in this document means that such features or functions are present in at least the described example, whether or not described as an example, and they may, but need not, be present in some or all other examples. Thus, "example", "for example", "may" or "might" refers to a specific instance of a class of examples. The attributes of an instance may be attributes of only that instance, or of the class, or of a subclass of the class, which includes some but not all instances in the class. Thus, features described with reference to one example but not with reference to another example are implicitly disclosed and may, where possible, be used as part of a working combination in that other example, but do not necessarily have to be used in that other example.

[0252] While an effort has been made to draw attention in the foregoing description to those features considered to be important, it should be understood that the applicant may seek protection by way of claims for any patentable feature or combination of features referenced above and / or shown in the accompanying drawings whether or not emphasis has been placed thereon.

Claims

1. An electronic device comprising components for: Simultaneously controlling content presented via a handheld portable device and a spatial audio device; providing, prior to an action by a user, via the spatial audio device, to the user a concurrent presentation of a first portion of the spatial audio content and a second portion of the spatial audio content; In response to the user's action, dividing the spatial audio content into at least the first part and the second part, wherein the content includes at least the spatial audio content; In response to the user's action, providing the first portion of the spatial audio content instead of the second portion for presentation to the user via the handheld portable device instead of the spatial audio device, wherein providing the first portion of the spatial audio content for presentation via the handheld portable device includes downmixing the first portion of the spatial audio content to produce a modified first portion of the spatial audio content.

2. The electronic device of claim 1 , comprising means for causing, in response to the action of the user, simultaneous presentation of the first portion of the spatial audio content via the handheld portable device and the second portion of the spatial audio content via the spatial audio device to the user.

3. The electronic device of claim 1, wherein the spatial audio content is received via the device from a communication network as a downlink communication channel from a remote person, and wherein the first portion of the spatial audio content may include a voice of the remote person.

4. An electronic device according to claim 1, wherein the device includes components for: communicating with the handheld portable device and / or the spatial audio device to determine when the action of the user has occurred, wherein the action of the user includes movement of the handheld portable device toward the head of the user. 5 . The electronic apparatus of claim 2 , wherein a spatial audio scene defined by the second portion of the spatial audio content is modified depending on a location of the handheld portable device.

6. The electronic device of claim 5 , wherein the first portion of the spatial audio content is associated with a first speech associated with a first ear of the user, the handheld portable device being positioned at the first ear of the user, and wherein the second portion of the spatial audio content is associated with at least a different second speech, wherein the spatial audio scene defined with the second portion of the spatial audio content is modified to place the second speech at a second ear of the user opposite the first ear.

7. The electronic device according to claim 1, comprising: Means for editing the second portion of the spatial audio content in response to movement of the handheld portable device.

8. An electronic device according to any preceding claim, wherein the first portion of the spatial audio content is associated with an incoming call, and the second portion of the spatial audio content is associated with at least a portion of the content presented via the spatial audio device, which at least a portion continues to be presented after receiving the incoming call.

9. The electronic device of any one of claims 1 to 5, comprising means for enabling selection of the first portion of the spatial audio content by the user using a first-person perspective mediated reality.

10. The electronic device of claim 9 , wherein the first portion of the spatial audio content is an audio object selected via a direction of user attention, or an audio scene selected via a direction of user attention, or wherein the first portion of the spatial audio content is a first moving audio object, wherein the first portion of the audio content is presented as if the first moving audio object is stationary.

11. A hand-portable device comprising a display and configured as, or accommodating, an apparatus as claimed in any preceding claim.

12. A method for presenting content, comprising: Simultaneously controlling content presented via a handheld portable device and a spatial audio device; providing, prior to an action by a user, via the spatial audio device, to the user a concurrent presentation of a first portion of the spatial audio content and a second portion of the spatial audio content; In response to the user's action, dividing the spatial audio content into at least the first part and the second part, wherein the content includes at least the spatial audio content; In response to the user's action, providing the first portion of the spatial audio content instead of the second portion for presentation to the user via the handheld portable device instead of the spatial audio device, wherein providing the first portion of the spatial audio content for presentation via the handheld portable device includes downmixing the first portion of the spatial audio content to produce a modified first portion of the spatial audio content.

13. A computer program which, when executed on one or more processors, implements: Simultaneously controlling content presented via a handheld portable device and a spatial audio device; providing, prior to an action by a user, via the spatial audio device, to the user a concurrent presentation of a first portion of the spatial audio content and a second portion of the spatial audio content; In response to the user's action, dividing the spatial audio content into at least the first part and the second part, wherein the content includes at least the spatial audio content; as well as In response to the user's action, providing the first portion of the spatial audio content instead of the second portion for presentation to the user via the handheld portable device instead of the spatial audio device, wherein providing the first portion of the spatial audio content for presentation via the handheld portable device includes downmixing the first portion of the spatial audio content to produce a modified first portion of the spatial audio content.

Images