Virtual Scene

Abstract

An apparatus, method, and computer program are described, comprising: determining an audio mode among a plurality of audio modes for presenting first audio to the user based on the user's position and / or movement within a virtual scene, wherein the first audio includes multiple audio tracks located at different positions within the virtual scene; and rendering the first audio to the user in the determined audio mode, wherein: in the first audio mode among the plurality of audio modes, the positions of the audio tracks within the virtual scene are fixed, and in a second audio mode among the plurality of audio modes, the positions of the audio tracks within the virtual scene move with the user.

Description

Technical Field

[0001] The present specification relates to virtual scenes, for example, to presenting audio to a user based on the user's virtual position within the virtual scene. Background Art

[0002] The virtual scene enables the user to move within the virtual scene and experience the content in different ways depending on the user's virtual position within the scene.There remains a need for alternative arrangements for providing and controlling audio content in a virtual scene. Summary of the invention

[0003] In a first aspect, the specification describes an apparatus that includes components for performing the following operations: based on the user's position and / or movement within a virtual scene (e.g., a virtual reality, mixed reality, or augmented reality scene, the virtual scene may be, for example, a three-dimensional virtual scene), determining one of a plurality of audio modes for presenting a first audio (e.g., volumetric audio) to the user, wherein the first audio includes a plurality of audio tracks located at different positions within the virtual scene; and rendering the first audio to the user in the determined audio mode, wherein: in a first audio mode of the plurality of audio modes, the position of the audio track within the virtual scene is fixed (e.g., so that the user can move relative to the audio track), and in a second audio mode of the plurality of audio modes (e.g., a so-called "user-centric" mode), the position of the audio track within the virtual scene moves with the user. In the second audio mode, the position of the audio track within the virtual scene may be fixed relative to the user. The first audio may be provided within a virtual scene about which the user may move (e.g., using 6-DoF movement, as discussed further below). The first audio mode may be a six-degree-of-freedom audio mode and / or the second audio mode may be a three-degree-of-freedom audio mode.

[0004] In some example embodiments, the component for determining one of the multiple audio modes may also be configured to determine the audio mode based on the user's position relative to the first area associated with the first audio in the virtual scene. The component for determining one of the multiple audio modes may also be configured to determine that the first audio mode is used to present the first audio when the user enters the first area from outside the first area or approaches the first area. Some example embodiments also include: when initiating the first audio mode, locating the first area associated with the first audio in the virtual scene so that the user is located at the center point of the above-mentioned first area.

[0005] The component for determining one of the plurality of audio modes may also be configured to: when the user approaches the edge of the first area from the inside of the area, determine that the second audio mode is used to present the first audio. The component may also be configured to perform: in the second audio mode, moving (e.g., gradually, such as to make the movement of the audio smooth) the first area and the audio track of the first audio as the user moves.

[0006] The component may also be configured to enable the user to exit the first area. Exiting the first area may cause the first audio to change.

[0007] The component for determining one of the plurality of audio modes may also be configured to perform: in the event that the user is determined to be stationary (e.g., if the user does not move for more than a threshold period of time—the threshold period of time may be variable, such as context-specific), determining that the first audio mode is used to present the first audio to the user. Thus, a transition from the second audio mode to the first audio mode may be achieved.

[0008] The component may also be configured to select the first audio from a plurality of candidate first audios based on determining that the user is located within a constellation of audio tracks of the selected first audio in the virtual scene.

[0009] The component may also be configured to perform: determining that the user has exited the constellation / the constellation of tracks of the first audio; and in response to the determination of the exit, stopping rendering the first audio to the user.

[0010] In some example embodiments, audio from different instruments may be assigned to different tracks of the first audio. For example, different musical instructions may be presented at different locations within the associated virtual space (e.g., enabling a "constellation of instruments" to be provided).

[0011] The above-mentioned components may include: at least one processor; and at least one memory including computer program code, the at least one memory and the computer program being configured to cause the operation of the apparatus together with the at least one processor.

[0012] In a second aspect, the specification describes a method, comprising: determining one of a plurality of audio modes for presenting first audio to a user based on a position and / or movement of the user within a virtual scene, wherein the first audio comprises a plurality of audio tracks located at different positions within the virtual scene; and rendering the first audio to the user in the determined audio mode, wherein: in a first audio mode of the plurality of audio modes, the position of the audio track within the virtual scene is fixed, and in a second audio mode of the plurality of audio modes, the position of the audio track within the virtual scene moves with the user. In the second audio mode, the position of the audio track within the virtual scene may be fixed relative to the user.

[0013] Some example embodiments further include determining an audio mode based on a position of a user within the virtual scene relative to a first area associated with the first audio. Determining an audio mode among multiple audio modes may include determining that the first audio mode is used to present the first audio when the user enters the first area or approaches the first area from outside the first area.

[0014] Some example embodiments further include: when initiating the first audio mode, locating a first area associated with the first audio within the virtual scene such that the user is located at a center point of the first area.

[0015] In some example embodiments, determining one of the plurality of audio modes further comprises: determining that a second audio mode is used to present the first audio when the user approaches an edge of the first region from within the region. Some example embodiments further comprise: in the second audio mode, moving (e.g., gradually moving) the first region and the audio track of the first audio as the user moves.

[0016] Some example embodiments further include enabling the user to exit the first area.

[0017] Some example embodiments further comprise determining, in an instance where the user is determined to be stationary, that a first audio mode is used to present the first audio to the user.

[0018] Some example embodiments further comprise selecting the first audio from a plurality of candidate first audios based on determining that the user is located within a constellation of audio tracks of the selected first audio in the virtual scene.

[0019] Some example embodiments further include: determining that a user has exited the constellation of tracks of the first audio; and in response to determining the exit, ceasing to render the first audio to the user.

[0020] In some example embodiments, audio from different instruments may be assigned to different tracks of the first audio.

[0021] In a third aspect, the specification describes an apparatus configured to perform any method described with reference to the second aspect.

[0022] In a fourth aspect, the present specification describes computer readable instructions which, when executed by a computing device, cause the computing device to perform any of the methods described with reference to the second aspect.

[0023] In a fifth aspect, the present specification describes a computer program comprising instructions for causing a device to perform at least the following operations: based on the user's position and / or movement within a virtual scene, determine an audio mode from multiple audio modes for presenting a first audio to a user, wherein the first audio comprises multiple audio tracks located at different positions within the virtual scene; and render the first audio to the user in the determined audio mode, wherein: in a first audio mode from the multiple audio modes, the position of the audio track within the virtual scene is fixed, and in a second audio mode from the multiple audio modes, the position of the audio track within the virtual scene moves with the user.

[0024] In a sixth aspect, the present specification describes a computer-readable medium (such as a non-transitory computer-readable medium) comprising program instructions stored thereon, the program instructions being used to perform at least the following operations: determining, based on the user's position and / or movement within a virtual scene, an audio mode from among multiple audio modes for presenting a first audio to a user, wherein the first audio comprises multiple audio tracks located at different positions within the virtual scene; and rendering the first audio to the user in the determined audio mode, wherein: in a first audio mode from among the multiple audio modes, the position of the audio track within the virtual scene is fixed, and in a second audio mode from among the multiple audio modes, the position of the audio track within the virtual scene moves with the user.

[0025] In a seventh aspect, the present specification describes a device comprising: at least one processor; and at least one memory comprising computer program code, which, when executed by the at least one processor, causes the device to: determine, based on the user's position and / or movement within a virtual scene, one of multiple audio modes for presenting a first audio to a user, wherein the first audio comprises multiple audio tracks located at different positions within the virtual scene; and render the first audio to the user in the determined audio mode, wherein: in a first audio mode among the multiple audio modes, the position of the audio track within the virtual scene is fixed, and in a second audio mode among the multiple audio modes, the position of the audio track within the virtual scene moves with the user.

[0026] In an eighth aspect, the present specification describes a device comprising: a component (such as a first processor, which may, for example, form part of a control system) for determining one of multiple audio modes for presenting a first audio to a user based on the user's position and / or movement within a virtual scene, wherein the first audio comprises multiple audio tracks located at different positions within the virtual scene; and a component (such as a rendering component, e.g., a media player) for rendering the first audio to the user in the determined audio mode, wherein: in a first audio mode among the multiple audio modes, the position of the audio track within the virtual scene is fixed, and in a second audio mode among the multiple audio modes, the position of the audio track within the virtual scene moves with the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Example embodiments will now be described, by way of example only, with reference to the following schematic drawings, in which:

[0028] Figure 1 is a block diagram of a virtual reality display system in which example embodiments may be implemented;

[0029] Figure 2 is a virtual environment according to an example embodiment;

[0030] Figure 3 and Figure 4 is a flow chart illustrating an algorithm according to an example embodiment;

[0031] Figures 5 to 7 is a virtual environment according to an example embodiment;

[0032] Figure 8 is a flow chart illustrating an algorithm according to an example embodiment;

[0033] Fig. 9 and Fig.10 is a virtual environment according to an example embodiment;

[0034] Fig.11 is a flow chart illustrating an algorithm according to an example embodiment;

[0035] Fig.12 is a virtual environment according to an example embodiment;

[0036] Fig.13 is a block diagram of a system according to an example embodiment;

[0037] Fig.14 is a block diagram of components of a system according to an example embodiment; and

[0038] Fig.15A and Fig. 15BTangible media storing computer readable codes, which when executed by a computer perform operations according to example embodiments, are shown, respectively, a removable nonvolatile memory unit and a compact disk (CD). DETAILED DESCRIPTION

[0039] The scope of protection sought by various embodiments of the present invention is defined by the independent claims. Embodiments and features described in the specification that do not belong to the scope of the independent claims, if any, are to be interpreted as examples useful for understanding various embodiments of the present invention.

[0040] In the specification and drawings, like reference numerals refer to like elements throughout.

[0041] Virtual reality (VR) can generally be understood as a rendered version of a visual and / or audio scene. The rendering can be designed to simulate the visual and auditory sensory stimulation of the real world so as to provide the user with a natural experience that is at least significantly consistent with their movement in the virtual scene, subject to the limitations defined by the content and / or application.

[0042] In many, but not necessarily all, cases, VR requires the user to wear a head mounted display (HMD) to completely replace the user's field of view with a simulated visual presentation, and requires the user to wear headphones to provide the user with simulated audio content to similarly completely replace the sound scene of the physical space. Some form of head tracking and general motion tracking of the user consuming the VR content may also be necessary. This allows the simulated visual and audio presentation to be updated to ensure that various scene components, such as items and sound sources, remain consistent with the user's movement from the user's perspective. Additional components for interacting with the virtual reality simulation, such as controls or other user interfaces (UIs), may be provided, but these are not absolutely necessary to provide the experience.

[0043] In some use cases, VR can be purely visual or purely audio virtual reality. For example, an audio-only VR experience can be associated with a new type of music listening or any other audio experience.

[0044] Augmented reality (AR) generally refers to providing additional information or artificially generated items or content to the user, which are at least significantly covered on the user's current real-world environmental stimulation. In some such cases, the enhanced content can at least partially replace the user's real-world content. The additional information or content can be visual and / or auditory. AR can have a pure visual or pure audio presentation. For example, a user can move in a city and receive audio guidance related to, for example, navigation, location-based advertising, and any other location-based information.

[0045] Mixed reality (MR) is sometimes considered a more advanced form of AR, in which at least some virtual elements are inserted into the physical scene so that they provide the illusion that these elements are part of the real scene, and behave accordingly. For audio content, or indeed for audio-only use cases, many applications of AR and MR can seem difficult to distinguish from each other to the user. However, the differences are not only in visual content, but can also be related to audio. For example, MR audio rendering may take into account local room reverberation, while AR audio rendering may not.

[0046] In 3D space, there are a total of six degrees of freedom (DoF), which define how a user can move within the space. This movement is divided into two categories: rotational movement and translational movement (each with three degrees of freedom). For simple VR experiences, rotational movement is sufficient, where the user can turn his / her head (pitch, yaw, and roll) to experience the space from a stationary point or along an automatic movement trajectory. Translational movement means that the user can also change the rendered position at his / her will, that is, move along the x, y, and z axes in Euclidean space. Free-view AR / VR experiences allow both rotational and translational movement. The terms 3DoF, 3DoF+, and 6DoF are often used to talk about various degrees of freedom and related experiences. 3DoF+ is between 3DoF and 6DoF, and usually allows some limited user movement. For example, a limited 6DoF implementation can be considered, where the user sits down but can tilt his / her head in all directions.

[0047] Figure 1 is a block diagram of a virtual reality display system in which example embodiments may be implemented, the virtual reality display system generally indicated by reference numeral 10. The virtual reality display system 10 includes user equipment in the form of a virtual reality headset 14 for displaying visual data and / or presenting audio data for a virtual reality space, and a virtual reality media player 12 for rendering the visual and / or audio data on the virtual reality headset 14. In some example embodiments, separate user controls (not shown) may be associated with the virtual reality display system, such as handheld controllers.

[0048] The virtual space, world, or environment may be a computer-generated version of a space (e.g., a captured real-world space) in which a user may be immersed. In some example embodiments, the virtual space or scene may be entirely computer-generated. Virtual reality headset 14 may be of any suitable type. Virtual reality headset 14 may be configured to provide virtual reality video and / or audio content data to the user. In this way, the user may be immersed in the virtual space.

[0049] In the example virtual reality display system 10, the virtual reality headset 14 receives virtual reality content data from the virtual reality media player 12. The virtual reality media player 12 may be part of a separate device that is connected to the virtual reality headset 14 through a wired or wireless connection. For example, the virtual reality media player 12 may include a game console or a PC (personal computer) configured to transmit visual data to the virtual reality headset 14.

[0050] Alternatively, the virtual reality media player 12 may form part of the virtual reality headset 14 .

[0051] The virtual reality media player 12 may include a mobile phone, smartphone, or tablet computer configured to provide content through its display. For example, the virtual reality media player 12 may be a touch screen device with a larger display on the main surface of the device, through which video content can be displayed. The virtual reality media player 12 may be inserted into a bracket of a virtual reality headset 14. Using such a virtual reality headset 14, the smartphone or tablet computer may display visual data, which is provided to the user's eyes via corresponding lenses in the virtual reality headset 14. For example, virtual reality audio may be presented through speakers integrated into the virtual reality headset 14 or headphones connected thereto. The virtual reality display system 10 may also include hardware configured to convert the device to operate as part of the virtual reality display system 10. Alternatively, the virtual reality media player 12 may be integrated into the virtual reality headset 14. The virtual reality media player 12 may be implemented in software. In some example embodiments, a device including virtual reality media player software is referred to as a virtual reality media player 12.

[0052] The virtual reality display system 10 may include a component for determining the spatial position of the user and / or the orientation of the user's head. This can be achieved by determining the spatial position and / or orientation of the virtual reality headset 14. Thus, over continuous time frames, a measure of motion can be calculated and stored. Such a component may include a portion of the virtual reality media player 12. Alternatively, the component may include a portion of the virtual reality headset 14. For example, the virtual reality headset 14 may incorporate a motion tracking sensor that may include one or more of a gyroscope, an accelerometer, and a structured light system. These sensors generate position data, from which the current field of view (FOV) may be determined, and the FOV may be updated as the user and thus the virtual reality headset 14 changes position and / or orientation. The virtual reality headset 14 may include two digital screens for displaying stereoscopic video images of the virtual world in front of the user's respective eyes, and two headphones, earpieces, or speakers for delivering audio. The example embodiments herein are not limited to a particular type of virtual reality headset 14.

[0053] In some example embodiments, the virtual reality display system 10 may use the above-described six degrees of freedom method to determine the spatial position and / or orientation of the user's head. These may include measurements of pitch, roll, and yaw, as well as translational motion along left-right, front-back, and up-down axes in Euclidean space.

[0054] The virtual reality display system 10 may be configured to display virtual reality content data to the virtual reality headset 14 based on the spatial position and / or orientation of the virtual reality headset. Detected changes in spatial position and / or orientation (i.e., a form of motion) may result in corresponding changes in visual and / or audio data to reflect the user's position or orientation changes relative to the space into which the visual data is projected. This allows the consumption of virtual reality content data by the user experiencing a 3D virtual reality environment.

[0055] In the context of a volumetric virtual reality space or world, a user's position relative to content provided in the volumetric virtual reality content can be detected, for example, so that the user can move freely around individual objects or groups of objects within a given virtual reality space or world, and can view and / or hear objects from different angles based on the rotation of their head.

[0056] The audio data may be provided to headphones provided as part of the virtual reality headset 14. The audio data may represent spatial audio source content. Spatial audio may refer to the directional rendering of audio in a virtual reality space or world, such that a detected change in the user's spatial position or their head orientation may result in a corresponding change in the spatial audio rendering to reflect a transformation of the space into which the reference spatial audio data is rendered.

[0057] Figure 2 2 is a virtual environment according to an example embodiment, which is generally indicated by reference numeral 20. The virtual environment 20 can be implemented using the virtual reality display system 10 described above. The virtual environment 20 shows a user 22 and first to third audio sources 24 to 26. The user 22 can wear the virtual reality headset 14 described above to experience the virtual environment 20.

[0058] The virtual environment 20 may thus present a virtual scene (e.g., a virtual reality, mixed reality, or augmented reality scene) to the user 22. The scene may, for example, be a three-dimensional virtual reality scene.

[0059] The virtual environment 20 is a virtual audio scene, and the user 22 has a position and orientation within the scene. The audio presented to the user 22 (e.g., using the virtual reality headset 14) depends on the position and orientation of the user 22, so that a 6DoF scene can be provided.

[0060] For example, the first audio source 24 to the third audio source 26 can be used to output songs to the user 22. In an example embodiment, different instruments (or audio from different instruments) are assigned to different audio sources (or tracks) 24 to 26 of the first audio. Thus, different instruments can be presented at different locations within the virtual environment 20 (e.g., enabling the provision of an "instrument constellation"). Such a constellation is particularly suitable for presenting traditional content, such as music tracks that can be used as a multi-track recording.

[0061] 6DoF audio provides an audio experience in which the user 22 can move in the audio scene. However, providing a full 6DoF content experience may require spatially arranging and creating content specifically for 6DoF consumption. Such a system may not be compatible with traditional audio content.

[0062] Figure 3 is a flow chart illustrating an algorithm, generally indicated by reference numeral 30 , according to an example embodiment.

[0063] The algorithm 30 begins at operation 32, where one of a plurality of audio modes is determined for presenting first audio (e.g., volumetric audio) to a user (e.g., user 22). The determination of the audio mode may be based on the user's position and / or movement within a virtual scene, such as virtual environment 20. Thus, the first audio is provided within a virtual scene in which the user may move (e.g., in 6-DoF motion). The first audio may include a plurality of audio tracks located at different locations within the virtual scene. Audio sources 24 to 26 are examples of such audio tracks.

[0064] At operation 34 , the audio is rendered (eg, to user 22 ) in the determined audio mode (ie, the mode determined in operation 32 ).

[0065] As described in detail below, the audio mode may include a first audio mode in which the position of the audio track within the virtual scene is fixed (so that the user can move relative to the audio track) and a second audio mode in which the position of the audio track within the virtual scene moves with the user. The first audio mode may be a 6-DoF audio mode. The second audio mode may be a 3-DoF audio mode.

[0066] Figure 4 is a flow chart illustrating an algorithm according to an example embodiment, which is generally indicated by reference numeral 40. Algorithm 40 is described below with reference to Figures 5 to 7 Further description.

[0067] The algorithm 40 begins at optional operation 42, where a user, such as user 22, enters an audio constellation, such as a constellation of audio sources or tracks. For example, the first audio source 24 to the third audio source 26 may constitute an audio constellation. For example, operation 42 may involve selecting a first audio from a plurality of candidate first audios by determining that the user is located within or near a track constellation of a selected first audio from the plurality of candidate first audios in a virtual scene.

[0068] In operation 44 of algorithm 40, an audio mode is determined.

[0069] Figure 5 1 is a virtual environment according to an example embodiment, which is generally indicated by reference numeral 50. The virtual environment 50 includes a user 22 and first to third audio sources 24 to 26. The virtual environment 50 also includes a first area 52. Figure 5 As shown, the user 22 is outside the first area 52. This area may be visible to the user in some manner within the virtual environment, such as objects having different colors depending on whether the user 22 is inside or outside the first area 52 (although this is not required for all example embodiments).

[0070] In operation 44 of algorithm 40, determination of the audio mode may depend on the location of user 22 relative to first area 52. For example, audio mode selection may depend on whether user 22 is outside first area 52 (as shown in virtual environment 50) or inside first area 52 (as described below).

[0071] Figure 6 is a virtual environment according to an example embodiment, which is generally indicated by reference numeral 60. The virtual environment 60 includes a user 22, first to third audio sources 24 to 26, and a first area 52. In the virtual environment 60, the user 22 approaches the first area from outside the area.

[0072] In operation 46 of algorithm 40, a first audio mode is initiated. Operation 46 may initiate the first audio mode when the user approaches (eg, contacts) first area 52 (as shown in virtual environment 60).

[0073] In operation 48 of algorithm 40 , when the first audio mode is initiated, the first area 52 is located in the virtual space such that the user 22 is at a center point of the first area.

[0074] Figure 770 is a virtual environment according to an exemplary embodiment, which is generally indicated by reference numeral 70. The virtual environment 70 includes the user 22, the first audio source 24 to the third audio source 26, and the first area 52. In the virtual environment 70, the first area 52 is positioned so that the user is at the center point of the area. Therefore, the virtual environment 70 shows the situation after the operation 48 of the algorithm 40 is started.

[0075] In the virtual environment 70 in which the user experiences the constellation including the first to third audio sources 24 to 26 in the first audio mode, the audio sources are presented to the user 22 with six degrees of freedom. Thus, the user is able to move while the positions of the audio sources remain fixed (so-called "world lock"). Thus, the user can experience the sound scene differently by moving relative to the audio sources 24 to 26 (and any other audio sources (not shown) that may form part of the audio constellation).

[0076] Figure 8 8 is a flow chart illustrating an algorithm according to an example embodiment, which is generally indicated by reference numeral 80. Algorithm 80 will be referred to below. Fig. 9 and Fig.10 Further description.

[0077] Algorithm 80 begins at operation 82 where an audio mode is determined.

[0078] Fig. 9 80 is a virtual environment in accordance with an example embodiment, which is generally indicated by reference numeral 90. The virtual environment 90 includes the user 22, the first audio source 24 to the third audio source 26, and the first area 52. In the virtual environment 90, the user 22 approaches or touches an edge or boundary of the first area 52 of the constellation. In response, it is determined in operation 82 of the algorithm 80 that the second audio mode should be entered.

[0079] At operation 84 of the algorithm 80, a second audio mode is initiated. In the second audio mode, the positions of the audio sources 24 to 26 of the first audio and the first area 52 move as the user moves. Therefore, the first audio constellation is not moved outside the first area 52, but moves so that the user stays within the first area. Therefore, compared to the 6DoF audio presentation of the first audio mode, the second audio mode presents the audio constellation to the user 22 as 3DoF audio.

[0080] At operation 86 of algorithm 80, the audio rendering to the user 22 is adjusted as the user moves.

[0081] Fig.10 1 is a virtual environment in accordance with an example embodiment, which is generally indicated by reference numeral 100. The virtual environment 100 includes a user 22, first to third audio sources 24, 26, and a first area 52.

[0082] like Fig.10 As shown, as the user 22 moves, the first area 52 moves from an initial position to a position 102, so that the user remains in the center of the first area. The audio constellation (including the first audio source 24 to the third audio source 26) and the position of the first area gradually move so that the user is eventually in the center of the constellation and / or the best listening position. This change can be gradual, so that the object slowly slides to the new position, so that the movement of the audio is smooth. Third-party applications (such as immersive phone calls) can trigger this feature to better allow the user to move. In an immersive call scenario, it may be beneficial for the call audio scene to follow the user. As another example, launching an application such as a motion tracker can trigger this feature.

[0083] Fig.11 is a flow chart illustrating an example algorithm according to an example embodiment, the algorithm being generally indicated by reference numeral 110. Algorithm 110 includes many of the features of algorithms 40 and 80 described above.

[0084] Algorithm 110 begins at operation 111, where a first audio mode is initiated (as in operation 46 described above). As described above, in the first audio mode, first audio including multiple audio tracks located at different locations within the virtual scene is rendered to the user. The locations of the audio tracks within the virtual scene are fixed. Therefore, the first audio is provided in a VR world in which the user can move (using 6 degrees of freedom movement).

[0085] At operation 112 of algorithm 110 , the user is placed at the center of the first area (and associated audio constellation), as in operation 48 described above.

[0086] At operation 114, it is determined whether the user is at (or near) the edge of the first area 52, for example Fig. 9 If not, the algorithm remains in the first audio mode (and the user 22 is able to move relative to the tracks of the relevant audio constellation). If the user is at (or near) the edge of the first area 52, the algorithm 110 moves to operation 115.

[0087] At operation 115, it is determined whether the user is exiting the track constellation of the first audio (as discussed further below). If so, the algorithm 110 moves to operation 119, where the algorithm is exited, resulting in the first audio being rendered to the user being stopped. If not, the algorithm 110 moves to operation 116 (discussed below). In an example embodiment, when the audio output (such as a song) presented to the user ends, the user automatically exits the constellation. The user can then enter another constellation and begin consuming audio within that constellation.

[0088] At operation 116, it has been determined that the user is at or near the boundary of the first area 52, and the user has not exited the area. In response, a second audio mode is initiated. For example, this behavior can be triggered by an external application (such as an immersive call). The user 22 can avoid this behavior through interaction, such as avoiding an object when exiting the first area. (This is a mechanism by which the user can exit the first area in the above operation 115.)

[0089] At operation 117 of algorithm 110, it is determined whether user 22 is stationary or moving. For example, it may be determined whether the user has been stationary for more than a threshold period of time. If user 22 is determined to be stationary, the algorithm moves to operation 111 (wherein the first audio mode is re-entered). Otherwise, algorithm 110 remains in the second audio mode. Note that the threshold period of time may be variable (e.g., context specific).

[0090] In operation 118 of algorithm 110, algorithm 110 is in a second operating mode. In the second mode, the constellation of tracks 24 to 26 is centered on the user so that the user can drag the content in 3DoF. The rendering of the tracks is adjusted. For example, the rendering can be gradually adjusted to make the motion smooth. In this way, the audio constellation can be gradually updated so that the user is eventually in the center and / or optimal listening position of the constellation. When the user is in the middle of the relevant audio constellation, the constellation automatically moves with the user (i.e., the audio is centered on the user). As described above, such a constellation is particularly suitable for presenting traditional content, such as music tracks that can be used as multi-track recordings. The method conveniently supports the user to listen to background music as the object constellation is automatically carried with them, so as to ensure a good listening experience even when in motion.

[0091] Operation 118 of algorithm 110 repeats (and the algorithm remains in the second audio mode) until, in an instance of operation 117 , the user is deemed to be stationary.

[0092] Fig.12 is a virtual environment in accordance with an example embodiment, generally indicated by reference numeral 120. Virtual environment 120 includes user 22, first through third audio sources 24-26, and first area 102. In virtual environment 120, user 22 has exited first area 102 (thereby implementing operation 119 of algorithm 110).

[0093] The user 22 may exit the first zone 102 through a defined interaction, such as avoiding an object when exiting the first zone 102. Once the user 22 is outside the first zone 102, a change in functionality may be visualized to the user, e.g., using an object of a different color. Alternatively or additionally, the user may exit the first zone 102 as a result of external factors, such as the end of a media item providing related audio content.

[0094] Fig.13 is a block diagram of a system, generally indicated by the reference number 130 , in accordance with an example embodiment.

[0095] System 130 is an example deployment of a 6DoF renderer for MPEG-I 6DoF audio.Those skilled in the art will be aware of alternative possible implementations of the principles described herein.

[0096] The MPEG-I 6DoF audio renderer 130 receives an encoder input format specification or other specification that describes a scene graph (e.g., scene geometry and object positions). Such an input scene may also define metadata of a content constellation.

[0097] Definitions used by system 130 include at least some of the following:

[0098] The position(s) of audio objects (such as audio sources 24 to 26);

[0099] The location of the first area 52 or 102;

[0100] A trigger condition for user locking of the audio constellation (second audio mode) when the user position matches the first area 52 or 102 and the user is centered in the constellation;

[0101] A trigger condition for world lock (first audio mode) of the audio constellation when the user position is within the first area 52 or 102;

[0102] • A trigger condition for world lock (first audio mode) of an audio constellation when the audio output (e.g., a music track) associated with the constellation object ends.

[0103] The triggers may be handled on the rendering side (such as blocks “Interaction Handling” and “Position and Pose Updates” in system 130 ).

[0104] In world lock mode (first audio mode), the position of audio objects relative to the user is updated taking into account user translation in Euclidean coordinates x, y, z and rotation in yaw, pitch, roll. In user lock mode (second audio mode), the position of objects relative to the user is updated only taking into account rotation of the user's head in yaw, pitch, and roll.

[0105] After the object positions are updated, the audio can be rendered from the correct direction and distance using head-related transfer function (HRFT) filtering and distance / gain attenuation. Virtual acoustics such as reverberation, adjusted according to the characteristics of the virtual space, can be used to enhance immersion. Those skilled in the art will be aware of alternative implementations of such functionality.

[0106] Many extensions or modifications to the MPEG-I Encoder Input Format (EIF) may be implemented to support aspects of the example embodiments described herein. These may include:

[0107] Expanded listener proximity conditions from world-locked positions to audio elements and audio element groups.

[0108] Channel source support in MPEG-I Audio EIF can be extended to support multiple object sources or HOA groups.

[0109] • EIF can be extended to trigger an update which is also the engagement duration. This may require attaching the constellation to the user at the beginning or disengagement later.

[0110] • The transition from one state to another (e.g., from one audio mode to another) can be immediate or continuous. When the user makes a move after reaching the edge of the inner circle, this can be extended to encompass a translation of the entire constellation.

[0111] • The listener condition is further extended from enabling from earlier disabling to modify the coordinate system from world-locked to user-relative.

[0112] Additionally, conditions utilize a mix of relative and world locking for both conditions and actions.

[0113] As example modifications to the encoder input format (EIF), the following modifications may be made to implement content region functionality for MPEG-I 6DoF audio content.

[0114] Modify the ObjectSource as follows:

[0115]

[0116]

[0117] Create a new entity called ContentZone:

[0118]

[0119] Different types of firstRenderingMode can be as follows:

[0120]

[0121]

[0122] For the sake of completeness, Fig.14is a schematic diagram of components of one or more of the previously described example embodiments, which are hereinafter collectively referred to as processing system 300. Processing system 300 may be, for example, an apparatus as recited in the following claims.

[0123] The processing system 300 may have a processor 302, a memory 304 tightly coupled to the processor and including a RAM 314 and a ROM 312, and an optional user input 310 and a display 318. The processing system 300 may include one or more network / device interfaces 308, which are used for connection to a network / device, for example, a wired or wireless modem. The network / device interface 308 may also operate as a connection with other devices, such as a device / device that is not a network-side device. Therefore, direct connection between devices / devices without network involvement is possible.

[0124] The processor 302 is connected to each of the other components so as to control the operation thereof.

[0125] The memory 304 may include non-volatile memory, such as a hard disk drive (HDD) or a solid state drive (SSD). The ROM 312 of the memory 304 stores, among other things, an operating system 315, and may store software applications 316. The RAM 314 of the memory 304 is used by the processor 302 to temporarily store data. The operating system 315 may contain code that implements aspects of the algorithms 30, 40, 80, and 110 when executed by the processor. Note that in the case of a small device / apparatus, the memory may be best suited for small size use, i.e., a hard disk drive (HDD) or a solid state drive (SSD) is not always used.

[0126] Processor 302 may take any suitable form. For example, it may be a microcontroller, multiple microcontrollers, a processor, or multiple processors.

[0127] The processing system 300 may be a stand-alone computer, server, console or a network thereof. The processing system 300 and the required structural parts may all be inside a device / device such as an IoT device / apparatus, i.e., embedded in a very small size.

[0128] In some example embodiments, the processing system 300 may also be associated with external software applications. These may be applications stored on a remote server device / apparatus and may run partially or exclusively on the remote server device / apparatus. These applications may be referred to as cloud-hosted applications. The processing system 300 may communicate with the remote server device / apparatus in order to utilize the software applications stored there.

[0129] Fig.15A and Fig. 15BTangible media storing computer readable code are shown, respectively a removable memory unit 365 and a compact disk (CD) 368, which can perform the method according to the above-described example embodiments when the computer is run. The removable memory unit 365 can be a memory stick, such as a USB memory stick, which has an internal memory 366 storing computer readable code. The internal memory 366 can be accessed by the computer system via a connector 367. The CD 368 can be a CD-ROM or DVD, etc. Other forms of tangible storage media can be used. The tangible medium can be any device / device capable of storing data / information that can be exchanged between devices / apparatus / networks.

[0130] Embodiments of the present invention may be implemented in software, hardware, application logic, or a combination of software, hardware, and application logic. The software, application logic, and / or hardware may reside in memory or on any computer medium. In an example embodiment, the application logic, software, or instruction set is maintained on any of a variety of conventional computer-readable media. In the context of this document, "memory" or "computer-readable medium" may be any non-transitory medium or device that can contain, store, convey, propagate, or transmit instructions for use by or in conjunction with an instruction execution system, device, or apparatus (such as a computer).

[0131] Where relevant, references to "computer-readable medium", "computer program product", "tangibly embodied computer program", etc., or "processor" or "processing circuit system", etc. should be understood to include not only computers with different architectures (such as single / multi-processor architectures and sequencer / parallel architectures), but also special-purpose circuits such as field-programmable gate arrays FPGAs, application-specific circuits ASICs, signal processing devices / apparatuses, and other devices / apparatuses. References to computer programs, instructions, codes, etc. should be understood to mean software for programmable processor firmware, programmable content of hardware devices / apparatuses such as instructions for a processor, or configured or configured settings for fixed-function devices / apparatuses, gate arrays, programmable logic devices / apparatuses, etc.

[0132] If desired, the different functions discussed herein may be performed in a different order and / or concurrently with each other. Furthermore, if desired, one or more of the above functions may be optional or may be combined. Similarly, it will also be understood that Figure 3 , Figure 4 , Figure 8 and Fig.11 The flowcharts and message sequences are merely examples, and the various operations described therein may be omitted, reordered, and / or combined.

[0133] It should be understood that the above exemplary embodiments are purely illustrative and do not limit the scope of the present invention.Other variations and modifications will be clear to those skilled in the art after reading this specification.

[0134] Furthermore, the disclosure of the present application should be understood to include any novel feature or any novel combination of features disclosed herein explicitly or implicitly, or any generalization thereof, and during the prosecution of the present application or any application derived therefrom, new claims may be formulated to cover any such feature and / or combination of such features.

[0135] Although various aspects of the invention are set out in the independent claims, further aspects of the invention comprise other combinations of features from the described example embodiments and / or the dependent claims with features of the independent claims, and not just the combinations explicitly set out in the claims.

[0136] It should also be noted here that although various examples are described above, these descriptions should not be viewed in a limiting sense. On the contrary, several variations and modifications may be made without departing from the scope of the present invention as defined in the appended claims.

Claims

1. An apparatus for processing audio, comprising means for performing the following: determining, based on a position and / or movement of a user relative to a first area associated with first audio within a virtual scene, one of a plurality of audio modes for presenting first audio to the user, wherein the first audio includes a plurality of audio tracks located at different locations within the virtual scene, and a second audio mode of the plurality of audio modes is determined when the user approaches an edge of the first area from within the first area; Rendering the first audio to the user in the determined audio mode, wherein: In a first audio mode of the plurality of audio modes, the plurality of audio tracks are fixed at the different positions within the virtual scene, and In the second audio mode among the multiple audio modes, the multiple audio tracks move with the user at the different positions within the virtual scene; and in the second audio mode, the first area and the multiple audio tracks of the first audio move with the movement of the user. 2 . The apparatus of claim 1 , wherein in the second audio mode, the plurality of audio tracks are fixed relative to the user at the different locations within the virtual scene.

3. The device according to claim 1, wherein the component for determining one of the multiple audio modes is further configured to: when the user enters the first area or approaches the first area from outside the first area, determine that the first audio mode is used to present the first audio. 4 . The apparatus according to claim 1 , wherein the component is further configured to perform: when the first audio mode is initiated, positioning the first area associated with the first audio within the virtual scene so that the user is at a center point of the first area. 5 . The apparatus according to claim 1 , wherein the component is further configured to perform: in the second audio mode, gradually moving the first area and the plurality of audio tracks of the first audio along with movement of the user. The apparatus according to claim 1 , wherein the component is further configured to enable the user to exit the first area.

7. The apparatus according to any one of claims 1 to 6, wherein the component for determining one of the multiple audio modes is further configured to perform: when the user is determined to be stationary, determining that the first audio mode is used to present the first audio to the user.

8. The apparatus according to any one of claims 1 to 6, wherein the component is further configured to: select the first audio from a plurality of candidate first audios based on determining that the user is located within a constellation of the audio track of the selected first audio in the virtual scene.

9. The apparatus according to any one of claims 1 to 6, wherein the component is further configured to perform: determining the constellation / constellations of tracks from which the user has exited the first audio; and In response to the determination of the exit, rendering of the first audio to the user is stopped.

10. The apparatus according to any one of claims 1 to 6, wherein the first audio mode is a six-degree-of-freedom audio mode and / or the second audio mode is a three-degree-of-freedom audio mode.

11. The apparatus of any one of claims 1 to 6, wherein audio from different instruments are assigned to different tracks of the first audio.

12. A method for processing audio, comprising: determining, based on a position and / or movement of a user relative to a first area associated with first audio within a virtual scene, one of a plurality of audio modes for presenting first audio to the user, wherein the first audio includes a plurality of audio tracks located at different locations within the virtual scene, and a second audio mode of the plurality of audio modes is determined when the user approaches an edge of the first area from within the first area; Rendering the first audio to the user in the determined audio mode, wherein: In a first audio mode of the plurality of audio modes, the plurality of audio tracks are fixed at the different positions within the virtual scene, and In the second audio mode among the multiple audio modes, the multiple audio tracks move with the user at the different positions in the virtual scene; and in the second audio mode, the first area and the multiple audio tracks of the first audio move with the movement of the user.

Images