Spatialized audio chat in a virtual metaverse

Abstract

Methods, systems, and computer-readable media for providing spatialized audio in a virtual experience are provided.SOLUTION: Spatialized audio may be used, for example, in audio communication such as voice and / or video chat. The chat may include spatialized audio that is combined at the client device or the online experience platform and directed to a particular user. Individual audio streams may be collected from multiple avatars and other objects and combined based on the target user. The audio may also include background and / or environmental sounds to provide a rich and immersive audio stream in the virtual experience.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Application No. 63 / 222,304, filed July 15, 2021, entitled SPATIALIZED AUDIO CHAT IN A VIRTUAL METAVERSE, the entire contents of which are incorporated herein by reference.

[0002] Embodiments relate generally to audio output via computing devices, and more particularly to methods, systems, and computer-readable media for providing spatialized audio in a virtual immersive environment, such as a metaverse place in a virtual metaverse. [Background technology]

[0003] Computer audio (e.g., chat between users of computing devices) often consists of mono or stereo audio being provided as the chat is received from a listening device or microphone. The provided audio is generally unfiltered or minimally filtered and may sound dry or direct, regardless of the actual virtual locations of the two avatars representing the users participating in the chat. Thus, as virtual experiences become more visually immersive, the simple nature of the provided audio can interfere with and / or detract from the immersive experience, e.g., by taking the user out of the experience.

[0004] The background art description provided herein is intended to provide a context for the present disclosure. The work of the named inventors to the extent described in this background art section, and aspects of the description that may not otherwise qualify as prior art at the time of filing, are not admitted, expressly or implicitly, as prior art to the present disclosure. Summary of the Invention [Means for solving the problem]

[0005] Implementations of this application relate to providing spatialized audio in the virtual metaverse.

[0006] According to one aspect, a computer-implemented method for spatialized audio in a virtual metaverse includes receiving a request to receive audio associated with a metaverse place in the virtual metaverse from a first user of a plurality of users, the first user being associated with a user device, the plurality of users being associated with respective avatars of a plurality of avatars in the metaverse place; retrieving a data model associated with the metaverse place, the data model including one or more spatial parameters representing one or more laws of physics that apply to the metaverse place; and extracting avatar information and scene information from the data model, the avatar information including position, velocity, and time of a plurality of avatars in the metaverse place, including the first avatar associated with the first user. a first avatar in the metaverse, wherein the scene information includes one or more of: a direction, a virtual occlusion, a reverberation, or a virtual wall virtually proximate to the first avatar in the metaverse; a second avatar in the metaverse, wherein the scene information includes one or more of: a direction, a virtual occlusion, a virtual wall, or a virtual wall virtually proximate to the first avatar in the metaverse; a third avatar in the metaverse, wherein the scene information includes one or more of: a direction, a virtual occlusion, a virtual wall, or a virtual wall virtually proximate to the first avatar in the metaverse; a fourth avatar in the metaverse, wherein the scene information includes one or more of: a direction, a virtual occlusion, a virtual wall, or a virtual wall virtually proximate to the first avatar in the metaverse; a fourth avatar in the metaverse, wherein the scene information includes one or more of: a direction, a virtual occlusion, a virtual wall, or a virtual wall virtually proximate to the first avatar in the metaverse; a fourth avatar in the metaverse, wherein the scene information includes one or more of: a direction, a virtual occlusion, a virtual wall, or a virtual wall; a fourth avatar in the metaverse, wherein the scene information includes one or more of: an occlusion, a virtual wall, or a virtual wall virtually proximate to the first avatar in the metaverse; a fourth avatar in the metaverse, wherein the scene information includes one or more of: an occlusion, a virtual wall, or a virtual wall ...

[0007] Various implementations of computer-implemented methods are described herein.

[0008] In some implementations, the spatial parameters include a distance attenuation parameter for attenuating audio based on the distance between avatars.

[0009] In some implementations, each audio stream received from each of the multiple users includes mono audio received at a microphone device, and the combined spatialized audio stream includes stereo audio.

[0010] In some implementations, the combined spatialized audio stream includes stereo audio generated by placing each user's mono audio at the location of their respective avatar.

[0011] In some implementations, the combined spatialized audio stream includes spatial audio based on audio streams received from users other than the first user among the plurality of users, and background audio, where the background audio is generated based on one or more of audio received from other users other than the first user and audio generated based on movement of an avatar within the metaverse.

[0012] In some implementations, the computer-implemented method further includes determining a set of prioritized audio streams received from each user of the plurality of users, and the step of transforming each audio stream further includes transforming the set of prioritized audio streams to create a spatialized audio stream.

[0013] In some implementations, the step of determining the set of prioritized audio streams includes prioritizing the audio streams received from each user of the plurality of users based on one or more of: proximity of the avatar within the metaverse place, speed of the avatar within the metaverse place, direction of the avatar within the metaverse place, virtual objects in proximity to the avatar within the metaverse place, capabilities of the user device, or user preferences of the first user.

[0014] In some implementations, audio streams associated with avatars closer to the receiving avatar are prioritized over audio streams associated with avatars farther from the receiving avatar, audio streams associated with avatars pointing towards the receiving avatar are prioritized over audio streams associated with avatars pointing away from the receiving avatar, and audio streams associated with avatars moving towards the receiving avatar are prioritized over audio streams associated with avatars moving away from the receiving avatar.

[0015] According to another aspect, a computer-implemented method for providing spatialized audio in a virtual metaverse is disclosed, the computer-implemented method including: receiving a request from a first user of a plurality of users to receive audio related to a metaverse place of the virtual metaverse, wherein the first user is associated with a user device and the plurality of users are associated with respective avatars of a plurality of avatars in the metaverse place; determining a set of prioritized audio streams received from each user of the plurality of users; converting the set of prioritized audio streams to create a spatialized audio stream; combining the spatialized audio streams to create a combined spatialized audio stream; and providing the combined spatialized audio stream to the user device.

[0016] Various implementations of computer-implemented methods are described herein.

[0017] In some implementations, the step of determining the set of prioritized audio streams includes prioritizing the audio streams received from each user of the plurality of users based on one or more of: proximity of the avatar within the metaverse place, speed of the avatar within the metaverse place, direction of the avatar within the metaverse place, virtual objects in proximity to the avatar within the metaverse place, capabilities of the user device, or user preferences of the first user.

[0018] In some implementations, audio streams associated with avatars closer to the receiving avatar are prioritized over audio streams associated with avatars farther from the receiving avatar, audio streams associated with avatars pointing towards the receiving avatar are prioritized over audio streams associated with avatars pointing away from the receiving avatar, and audio streams associated with avatars moving towards the receiving avatar are prioritized over audio streams associated with avatars moving away from the receiving avatar.

[0019] According to another aspect, a method includes: receiving a request from a first user of a plurality of users to receive audio associated with a metaverse place of a virtual metaverse, the first user being associated with a user device, the plurality of users being associated with respective avatars of a plurality of avatars in the metaverse place; retrieving a data model associated with the metaverse place, the data model including one or more spatial parameters representing one or more laws of physics that apply to the metaverse place; and extracting avatar information and scene information from the data model, the avatar information being transmitted to the first user. A system is disclosed that performs operations including extracting scene information including one or more of positions, velocities, or orientations of a plurality of avatars in a metaverse place including an associated first avatar, and scene information including one or more of occlusions, reverberations, or virtual walls virtually proximate to the first avatar in the metaverse place; transforming respective audio streams received from each of a plurality of users based on the avatar information and the scene information to create a spatialized audio stream, transforming one or more audio characteristics of at least one of the respective audio streams based on one or more spatial parameters; combining the spatialized audio streams to create a combined spatialized audio stream; and providing the combined spatialized audio stream to a user device.

[0020] Various implementations of the system are described herein.

[0021] In some implementations, the spatial parameters include a distance attenuation parameter for attenuating audio based on the distance between avatars.

[0022] In some implementations, each audio stream received from each of the multiple users includes mono audio received at a microphone device, and the combined spatialized audio stream includes stereo audio.

[0023] In some implementations, the combined spatialized audio stream includes stereo audio generated by placing each user's mono audio at the location of their respective avatar.

[0024] In some implementations, the combined spatialized audio stream includes spatial audio based on audio streams received from users other than the first user among the plurality of users, and background audio, where the background audio is generated based on one or more of audio received from other users other than the first user and audio generated based on movement of an avatar within the metaverse.

[0025] In some implementations, the operations further include determining a set of prioritized audio streams received from each user of the plurality of users, and transforming the respective audio streams further includes transforming the set of prioritized audio streams to create a spatialized audio stream.

[0026] In some implementations, determining the set of prioritized audio streams includes prioritizing the audio streams received from each user of the plurality of users based on one or more of: proximity of the avatar within the metaverse place, speed of the avatar within the metaverse place, direction of the avatar within the metaverse place, virtual objects in proximity to the avatar within the metaverse place, capabilities of the user device, or user preferences of the first user.

[0027] In some implementations, audio streams associated with avatars closer to the receiving avatar are prioritized over audio streams associated with avatars further away from the receiving avatar, and audio streams associated with avatars directed towards the receiving avatar are prioritized over associated audio streams.

[0028] In some implementations, the system further includes a spatialized audio manager configured to convert each audio stream received from each of the plurality of users, and an audio device override module configured to disable non-spatialized audio at the user device before providing the combined spatialized audio stream to the user device.

[0029] According to another aspect, a non-transitory computer-readable medium is provided that, in response to execution by a processing device, causes the processing device to retrieve a data model associated with a metaverse place of a virtual metaverse, the data model including one or more spatial parameters representing a group of physical laws that apply to the metaverse place; receive a request to join the metaverse place from a first user of a plurality of users, the first user being associated with a first avatar and a user device, and the plurality of users being associated with a plurality of avatars in the metaverse place; and extract avatar information and scene information from the data model in response to the request, the avatar information including position, velocity, or time of the first avatar and the plurality of avatars in the metaverse place. a first avatar and a second avatar based on the scene information, the first avatar being a virtual avatar; a second avatar and a third avatar being a virtual avatar; a third avatar and a fourth avatar being a virtual avatar; a fourth avatar and a fourth avatar being a virtual avatar; a fourth avatar and a fourth avatar being a virtual avatar; a second avatar and a third avatar being a virtual avatar; a fourth avatar and a fourth avatar being a virtual avatar; a fourth avatar and a fourth avatar being a virtual avatar; a fourth avatar and a fourth avatar being a virtual avatar; a fourth avatar and a fourth avatar being a virtual avatar; a fourth avatar and a fourth avatar being a virtual avatar; a fourth avatar and a fourth avatar being a virtual avatar; a fifth ...

[0030] Various implementations of non-transitory computer-readable media are described herein.

[0031] According to yet other aspects, portions, features, and implementation details of the systems, methods, and non-transitory computer-readable media may be combined to form additional aspects, including some aspects that omit and / or modify some or some of the individual components or features, include additional components or features, and / or other modifications, and all such modifications are within the scope of the present disclosure. [Brief explanation of the drawings]

[0032] [Figure 1] FIG. 1 illustrates an exemplary network environment for providing spatialized audio chat in a virtual metaverse, according to some implementations. [Figure 2A] FIG. 1 illustrates an exemplary network environment for providing spatialized audio chat in a virtual metaverse, according to some implementations. [Figure 2B] FIG. 1 illustrates an exemplary network environment for providing spatialized audio chat in a virtual metaverse, according to some implementations. [Figure 3] FIG. 1 illustrates an exemplary network environment for prioritizing spatialized audio streams in a virtual metaverse, according to some implementations. [Figure 4] FIG. 1 illustrates an exemplary three-dimensional metaverse place within a virtual experience, according to some implementations. [Figure 5] FIG. 1 illustrates an exemplary three-dimensional metaverse place within a virtual experience, according to some implementations. [Figure 6] 1 is a flow diagram of an example method for providing spatialized audio chat in a virtual metaverse, according to some implementations. [Figure 7] 1 is a flow diagram of an example method for prioritizing spatialized audio streams in a virtual metaverse, according to some implementations. [Figure 8] FIG. 1 is a block diagram illustrating an example computing device that may be used to implement one or more features described herein, according to some implementations. DETAILED DESCRIPTION OF THE INVENTION

[0033] One or more implementations described herein relate to spatialized audio associated with an online gaming platform. Features may include automatically prioritizing spatialized audio streams and providing spatialized audio based on position, velocity, and / or other factors associated with virtual objects, avatars, and other items within a metaverse place of a virtual metaverse.

[0034] The features described herein provide spatialized audio for output at a client device connected to an online platform, such as, for example, an online experience platform or an online gaming platform. The online platform may provide a virtual metaverse having multiple metaverse places associated therewith. A virtual avatar associated with a user can navigate the metaverse places and interact with the metaverse places, as well as with items, characters, other avatars, and objects within the metaverse places. The avatar can move from one metaverse place to another while experiencing spatialized audio, providing a more immersive and enjoyable experience. Spatialized audio streams from multiple users (e.g., or avatars associated with multiple users) can be prioritized based on many factors so that rich audio can be provided, taking into account the positions, speeds, movements, and actions of the avatars and characters, as well as the bandwidth, processing, and other capabilities of the client device.

[0035] By prioritizing and combining different audio streams, a combined spatialized audio stream can be provided for output at the client device, which provides a rich user experience, reduced computational effort for providing spatialized audio, and reduced bandwidth without compromising the virtual immersive experience. Additionally, a spatial audio application programming interface (API) is defined that enables users and developers to implement spatialized audio for almost any online experience, thereby enabling the creation of high-quality online virtual experiences, games, metaverses, and other interactions with immersive audio while requiring less technical proficiency from users and developers.

[0036] Online experience platforms and online gaming platforms (also called "user-generated content platforms" or "user-generated content systems") provide various ways for users to interact with each other. For example, users of an online experience platform may create games or other content or resources (e.g., characters, graphics, items for gameplay and / or for use in the virtual metaverse, etc.) within the online platform.

[0037] Users of an online experience platform may collaborate toward a common goal in a metaverse place, game, or game creation; share various virtual items (e.g., inventory items, game items, etc.); engage in audio chat (e.g., spatialized audio chat); send electronic messages to one another; and the like. Users of an online experience platform may interact with other users, for example, playing games involving characters (avatars) or other game objects and mechanisms. An online experience platform may also enable users of the platform to communicate with one another. For example, users of an online experience platform may communicate with one another using voice messages (e.g., via voice chat with spatialized audio), text messaging, video messaging (e.g., including spatialized audio), or a combination of the above. Some online experience platforms may provide a virtual three-dimensional environment, or multiple environments linked in the metaverse, in which users can interact with one another or play online games.

[0038] To help enhance the entertainment value of the online experience platform, the platform can provide rich audio for playback on user devices. The audio may include, for example, different audio streams from different users and background audio. According to various implementations described herein, the different audio streams may be converted into a spatialized audio stream. The spatialized audio streams may be combined, for example, to provide a combined spatialized audio stream for playback on the client device. Furthermore, prioritized audio streams may be provided so that bandwidth is reduced while still providing immersive spatialized audio. Furthermore, background audio streams may be combined with spatialized audio so that realistic background noises / effects are also played back to the user. Furthermore, characteristics of the metaverse, such as the surrounding medium (e.g., air, water, etc.), reverberation, reflections, hole size, wall density, ceiling height, doorways, hallways, object placement, non-player objects / characters, and other characteristics, are utilized to create the spatialized audio and / or background audio to enhance realism and immersion within the online virtual experience.

[0039] Figures 1 to 3: System architecture 1 illustrates an exemplary network environment 100 according to some implementations of the present disclosure. Network environment 100 (also referred to herein as a “system”) includes an online experience platform 102, a first client device 110, and a second client device 116 (collectively referred to herein as “client devices 110 / 116”), all connected via a network 122. Online experience platform 102 may include, among other things, a game engine 104, one or more games 105, a spatialized audio API 106, and a data store 108. Client device 110 may include a game application 112, and client device 116 may include a game application 118. Users 114 and 120 may use client devices 110 and 116, respectively, to interact with online experience platform 102 and other users utilizing online experience platform 102.

[0040] Network environment 100 is provided for illustrative purposes. In some implementations, network environment 100 may include the same, fewer, more, or different elements configured in the same or different manner as shown in FIG.

[0041] In some implementations, the network 122 may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wired network (e.g., an Ethernet network), a wireless network (e.g., an 802.11 network, a Wi-Fi network, or a wireless LAN (WLAN)), a cellular network (e.g., a Long Term Evolution (LTE) network), a router, a hub, a switch, a server computer, or a combination thereof.

[0042] In some implementations, the data store 108 may be non-transitory computer-readable memory (e.g., random access memory), a cache, a drive (e.g., a hard drive), a flash drive, a database system, or another type of component or device capable of storing data. The data store 108 may also include multiple storage components (e.g., multiple drives or multiple databases), possibly spread across multiple computing devices (e.g., multiple server computers).

[0043] In some implementations, the online experience platform 102 may include a server having one or more computing devices (e.g., a cloud computing system, a rack-mounted server, a server computer, a cluster of physical servers, a virtual server, etc.) In some implementations, the server may be included in the online experience platform 102, may be a separate system, or may be part of another system or platform.

[0044] In some implementations, the online experience platform 102 may include one or more computing devices (such as a rack-mounted server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer), data stores (e.g., hard disks, memory, databases), networks, software components, and / or hardware components that may be used to execute operations on the online experience platform 102 and provide users with access to the online experience platform 102. The online experience platform 102 may also include websites (e.g., one or more web pages) or application backend software that may be used to provide users with access to content provided by the online experience platform 102. For example, a user 114 / 120 may access the online experience platform 102 using a gaming application 112 / 118 on a client device 110 / 116.

[0045] In some implementations, the online experience platform 102 may include some type of social network that provides connections between users, or some type of user-generated content system that allows users (e.g., end users or consumers) to communicate with other users via the online experience platform 102, where the communication may include voice chat (e.g., synchronous and / or asynchronous voice communication with or without spatialized audio), video chat (e.g., synchronous and / or asynchronous video communication with or without spatialized audio), or text chat (e.g., synchronous and / or asynchronous text-based communication).

[0046] In some implementations of the present disclosure, a "user" may be represented as a single individual. However, other implementations of the present disclosure encompass a "user" (e.g., a creating user) being a set of users or an entity controlled by an automated source. For example, a set of individual users federated as a community or group within a user-generated content system may be considered a "user."

[0047] In some implementations, the online experience platform 102 may be a virtual gaming platform. For example, the gaming platform may offer single-player or multiplayer games to a community of users who may access or interact with games (e.g., user-generated games or other games) using client devices 110 / 116 over the network 122. In some implementations, the games (also referred to herein as “video games,” “online games,” “metaverse places,” or “virtual experiences”) may be, for example, two-dimensional (2D) games, three-dimensional (3D) games (e.g., 3D user-generated games), virtual reality (VR) games, or augmented reality (AR) games. In some implementations, users may search for games and game items and participate in gameplay with other users in one or more games. In some implementations, games may be played in real time with other users of the game. Similarly, some users may participate in real-time voice or video chat with other users of the game. As described herein, the real-time voice or video chat may include spatialized audio.

[0048] In some implementations, other collaboration platforms may be used with the features described herein instead of or in addition to the online experience platform 102 and / or the spatialized audio API 106. For example, social networking platforms, purchasing platforms, messaging platforms, creation platforms, etc. may be used with spatial audio features to provide immersive, spatialized audio to users outside of the game.

[0049] In some implementations, gameplay may refer to the interactions of one or more players using client devices (e.g., 110 and / or 116) within a game (e.g., 105), or the representation of those interactions on a display or other output device of client device 110 or 116. In some implementations, gameplay may instead refer to interactions within a virtual experience or metaverse place, and may include objectives that are similar to, different from, or the same as some games. Additionally, while referred to as "players," the terms "avatars," "users," and / or other terms may be used to refer to users who participate in and / or interact with an online virtual experience.

[0050] One or more games 105 are provided by the online experience platform. In some implementations, the games 105 may include electronic files that can be executed or loaded using software, firmware, or hardware configured to present game content (e.g., digital media items) to entities. In some implementations, the game applications 112 / 118 may execute in association with the game engine 104, and the games 105 may be rendered in association with the game engine 104. In some implementations, the games 105 may have a common set of rules or a common goal, and the virtual environments of the games 105 share a common set of rules or a common goal. In some implementations, different games may have different rules or goals from one another. It is noted that while specifically referred to as "games" or game-related, the game applications 112 / 118, games 105, and game engine 104 may also be referred to as virtual experience applications 112 / 118, virtual experiences 105, and / or virtual experience engines 104.

[0051] In some implementations, a game and / or virtual experience may have one or more environments (also referred to herein as a “game environment,” “metaverse place,” or “virtual environment”), and multiple environments may be linked. An example of an environment may be a three-dimensional (3D) environment. One or more environments of a game 105 or virtual experience may be collectively referred to herein as a “world,” “game world,” “virtual world,” “universe,” or “metaverse.” An example of a world may be a 3D metaverse place of a game 105. For example, a user may build a metaverse place that is linked to another metaverse place created by another user different from the first user. Characters of a virtual experience may cross virtual boundaries to enter adjacent metaverse places. Furthermore, sounds, theme music, and / or background music may also cross virtual boundaries, such that an avatar standing near a virtual boundary may hear spatialized audio that includes at least some sounds emanating from the adjacent metaverse place. In this manner, spatialized audio may enable a fully immersive experience, including virtual audio that represents a similarity of sound propagation with real-world environments.

[0052] It may be noted that a 3D environment or world uses graphics that use three-dimensional representations of geometric data representing the content (or at least present the content to appear as 3D content, regardless of whether 3D representations of geometric data are used), whereas a 2D environment or world uses graphics that use two-dimensional representations of geometric data representing the game content.

[0053] In some implementations, the online experience platform 102 may host one or more games 105 and allow users to interact with the games 105 (e.g., search for games, game-related content, or other content) using game applications 112 / 118 on client devices 110 / 116. Users (e.g., 114 and / or 120) of the online experience platform 102 may play, create, interact with, or build games 105, search games 105, communicate with other users, create, build, and / or search for objects (e.g., also referred to herein as “items” or “game objects” or “virtual game items”) for the games 105. For example, when generating user-generated virtual items, users may, among other things, create characters, decorations for characters, one or more virtual environments for an interactive game, or build structures used within the games 105.

[0054] In some implementations, users may buy, sell, or trade virtual game objects of a game, such as in-platform currency (e.g., virtual currency), with other users of the online experience platform 102. In some implementations, the online experience platform 102 may transmit game content to a game application (e.g., 112). In some implementations, game content (also referred to herein as "content") may refer to any data or software instructions (e.g., game objects, games, user information, video, images, commands, media items, etc.) related to the online experience platform 102 or a game application.

[0055] In some implementations, a game object (e.g., also referred to herein as an "item" or "object" or "virtual game item") may refer to an object used, created, shared, or otherwise depicted in a game application 105 of the online experience platform 102 or a game application 112 or 118 of a client device 110 / 116. For example, a game object may include parts, models, characters, tools, weapons, clothing, buildings, vehicles, currency, flora, fauna, components of the above (e.g., windows of a building), etc.

[0056] It may be noted that the online experience platform 102 hosting the game 105 is provided for purposes of example and not limitation. In some implementations, the online experience platform 102 may host one or more media items, which may include communication messages from one user to one or more other users. Media items may include, but are not limited to, digital videos, digital movies, digital photos, digital music, audio content, melodies, website content, social media updates, e-books, e-magazines, digital newspapers, digital audiobooks, e-journals, weblogs, real simple syndication (RSS) feeds, e-comics, software applications, etc. In some implementations, media items may be electronic files that can be executed or loaded using software, firmware, or hardware configured to present the digital media items to entities.

[0057] In some implementations, a game 105 may be associated with a particular user or a particular group of users (e.g., a private game) or may be made generally available to users of the online experience platform 102 (e.g., a public game). In some implementations in which the online experience platform 102 associates one or more games 105 with a particular user or group of users, the online experience platform 102 may use user account information (e.g., a user account identifier such as a username and password) to associate a particular user with a game 105. Similarly, in some implementations, the online experience platform 102 may use developer account information (e.g., a developer account identifier such as a username and password) to associate a particular developer or group of developers with a game 105.

[0058] In some implementations, the online experience platform 102 or the client device 110 / 116 may include a game engine 104 or a game application 112 / 118. The game engine 104 may include a game application similar to the game application 112 / 118. In some implementations, the game engine 104 may be used to deploy or run a game 105. For example, the game engine 104 may include, among other features, a rendering engine (“renderer”) for 2D, 3D, VR, or AR graphics, a physics engine, a collision detection engine (and collision response), a sound engine, a spatialization audio manager / engine, an audio mixer, an audio subscription exchange, an audio subscription logic, an audio subscription prioritizer, a real-time communications engine, scripting capabilities, an animation engine, an artificial intelligence engine, networking capabilities, streaming capabilities, memory management capabilities, threading capabilities, scene graph capabilities, or video support for cinematics. Components of the game engine 104 may generate commands that help compute and render the game (e.g., rendering commands, collision commands, physics commands, etc.) and may convert audio (e.g., converting mono or stereo sound into a spatialized audio stream, etc.). In some implementations, the game applications 112 / 118 of the client devices 110 / 116 may each work independently, in cooperation with the game engine 104 of the online experience platform 102, or a combination of both.

[0059] In some implementations, both the online experience platform 102 and the client devices 110 / 116 run game engines (104, 112, and 118, respectively). The online experience platform 102, using the game engine 104, may perform some or all of the game engine functionality (e.g., generate physics commands, rendering commands, spatialized audio commands, etc.) or may offload some or all of the game engine functionality to the game engine 104 of the client device 110. In some implementations, each game 105 may have a different ratio between the game engine functionality running on the online experience platform 102 and the game engine functionality running on the client devices 110 and 116.

[0060] For example, the game engine 104 of the online experience platform 102 may be used to generate physics commands when there is a collision between at least two game objects, while further game engine functionality (e.g., generating rendering commands or combining spatialized audio streams) may be offloaded to the client device 110. In some implementations, the ratio of game engine functionality executed on the online experience platform 102 to game engine functionality executed on the client device 110 may be changed (e.g., dynamically) based on gameplay conditions. For example, if the number of users participating in gameplay of the game 105 exceeds a threshold number, the online experience platform 102 may execute one or more game engine functions previously executed by the client device 110 or 116.

[0061] For example, a user may be playing a game 105 on client devices 110 and 116 and may send control instructions (e.g., user inputs such as right, left, up, down, etc., user selections, or character position and velocity information) to the online gaming platform 102. After receiving the control instructions from client devices 110 and 116, the online experience platform 102 may send gameplay instructions (e.g., position and velocity information for characters participating in group gameplay, or commands such as rendering commands, collision commands, spatialized audio commands, etc.) to the client devices 110 and 116 based on the control instructions. For example, the online experience platform 102 may perform one or more logical operations (e.g., using the game engine 104) in accordance with the control instructions to generate gameplay instructions for the client devices 110 and 116. In other cases, the online experience platform 102 may pass one or more of the control instructions from one client device 110 to other client devices (e.g., 116) participating in the game 105. Client devices 110 and 116 may use the gameplay instructions to render gameplay for presentation on the displays of client devices 110 and 116. Client devices 110 and 116 may also use the gameplay instructions to create, modify, and / or combine spatialized audio streams for output at audio output devices of client devices 110 and 116.

[0062] In some implementations, control instructions may refer to instructions indicating in-game actions of a user's character. For example, control instructions may include user input, user selection, gyroscope position and orientation data, force sensor data, etc. for controlling in-game actions such as right, left, up, down, etc. The control instructions may include character position and velocity information. In some implementations, control instructions are sent directly to the online experience platform 102. In other implementations, control instructions may be sent from the client device 110 to another client device (e.g., 116), which generates gameplay instructions using a local game engine 104. The control instructions may include instructions to play voice communication messages or other sounds from another user on an audio device (e.g., speakers, headphones, etc.).

[0063] In some implementations, gameplay instructions may refer to instructions that enable client device 110 (or 116) to render gameplay of a game, such as a multiplayer game. Gameplay instructions may include one or more of user input (e.g., control instructions), character position and velocity information, or commands (e.g., physics commands, rendering commands, collision commands, etc.). As described in more detail herein, character position and velocity information may be used to determine an appropriate head-related transfer function (HRTF) associated with another character so that a spatialized audio stream representing sound propagation in the real world may be created for that character. Associated HRTFs, position information, velocity information, Baum-Welch (BW) algorithm data, virtual auditory display (VAD) data, and / or other data may be stored by online experience platform 102 in data store 108.

[0064] In some implementations, characters (or broadly, game objects) are constructed from components that combine automatically to assist the user in editing, one or more of which may be selected by the user. One or more characters (also referred to herein as “avatars” or “models”) may be associated with a user, who may control the character to facilitate the user's interaction with the game 105. In some implementations, a character may include components such as body parts (e.g., head, hair, arms, legs, etc.) and accessories (e.g., T-shirt, glasses, decorative images, tools, etc.). In some implementations, customizable character body parts include, among others, head type, body part type (arms, legs, torso, and hands), face type, hair type, and skin type. In some implementations, customizable accessories include clothing (e.g., shirt, pants, hat, shoes, glasses, etc.), weapons, or other tools.

[0065] In some implementations, the user may also control the scale of the character (e.g., height, width, or depth) or the scale of components of the character. In some implementations, the user may control the proportions of the character (e.g., blocky, anatomical, etc.). It may be noted that in some implementations, a character may not include a character game object (e.g., body parts, etc.), but the user may control the character (without a character game object) to facilitate the user's interaction with the game (e.g., a puzzle game where there is no rendered character game object, but the user still controls a character to control in-game action).

[0066] In some implementations, components such as body parts may be basic geometric shapes such as blocks, cylinders, spheres, or some other basic shape such as a wedge, ring, tube, channel, etc. In some implementations, a creator module may publish a user's character for viewing or use by other users of the online experience platform 102. In some implementations, creating, modifying, or customizing a character, other game object, game 105, or game environment may be performed by a user using a user interface (e.g., a developer interface), with or without scripting (or with or without an application programming interface (API)). It may be noted that, for purposes of illustration and not limitation, characters are described as having the form of a humanoid robot. It may further be noted that characters may have any form, such as a vehicle, an animal, an inanimate object, or other creative form.

[0067] In some implementations, the online experience platform 102 may store characters created by a user in the data store 108. In some implementations, the online experience platform 102 maintains a character catalog and a game catalog that may be presented to a user via the game engine 104, the game 105, and / or the client device 110 / 116. In some implementations, the game catalog includes images of games stored in the online experience platform 102. In addition, a user may select a character (e.g., a character created by the user or another user) from the character catalog to participate in a selected game. The character catalog includes images of characters stored in the online experience platform 102. In some implementations, one or more of the characters in the character catalog may have been created or customized by the user. In some implementations, the selected character may have a character setting that defines one or more of the character's components.

[0068] In some implementations, a user's character may include a configuration of components, and the configuration and appearance of the components, as well as the character's appearance more generally, may be defined by a character setting. In some implementations, a user's character's character setting may be selected, at least in part, by the user. In other implementations, a user may select a character having a default character setting or another user-selected character setting. For example, a user may select a default character from a character catalog having a predefined character setting, and further, the user may customize the default character by changing parts of the character setting (e.g., adding a shirt with a customized logo). A character setting may be associated with a particular character by the online experience platform 102.

[0069] In some implementations, client device 110 or 116 may include a computing device, such as a personal computer (PC), a mobile device (e.g., a laptop, a mobile phone, a smartphone, a tablet computer, or a netbook computer), a network-connected television, a game console, or the like. In some implementations, client device 110 or 116 may also be referred to as a "user device." In some implementations, one or more client devices 110 or 116 may connect to online experience platform 102 at any time. It may be noted that the number of client devices 110 or 116 is given by way of example and not limitation. In some implementations, any number of client devices 110 or 116 may be used.

[0070] In some implementations, each client device 110 or 116 may include an instance of a gaming application 112 or 118, respectively. In one implementation, the gaming application 112 or 118 may enable a user to use and interact with the online experience platform 102, such as searching for games, experiences, or other content; controlling virtual characters in virtual experiences hosted by the online experience platform 102; or viewing or uploading content such as games 105, images, video items, web pages, documents, etc. In one example, the gaming application may be a web application (e.g., an application that operates in conjunction with a web browser) that can access, retrieve, present, or navigate content provided by a web server (e.g., virtual characters in a virtual environment, etc.). In another example, the gaming application may be a native application (e.g., a mobile application, app, or game program) that is installed and executed locally on the client device 110 or 116 and enables a user to interact with the online experience platform 102. The game application may render, display, or present content (e.g., web pages, user interfaces, media viewers, audio streams) to the user. In implementations, the game application may also include an embedded media player that is embedded in a web page.

[0071] According to aspects of the present disclosure, the game application 112 / 118 may be an online experience platform application through which a user builds, creates, edits, and uploads content to the online experience platform 102 and interacts with the online experience platform 102 (e.g., plays games 105 hosted by the online experience platform 102). Thus, the game application 112 / 118 may be provided to the client device 110 or 116 by the online experience platform 102. In another example, the game application 112 / 118 may be an application that is downloaded from a server.

[0072] In some implementations, a user may log in to the online experience platform 102 through a game application. The user may access a user account by providing user account information (e.g., a username and password), and the user account is associated with one or more characters that can be used to participate in one or more games 105 of the online experience platform 102.

[0073] Generally, functions described as being performed by online experience platform 102 may also be performed by client device 110 or 116 or a server, as appropriate, in other implementations. In addition, functions attributed to a particular component may be performed by different or multiple components working together. Online experience platform 102 may also be accessed as a service offered to other systems or devices through an appropriate application programming interface (API), and thus is not limited to use in websites.

[0074] In some implementations, the online experience platform 102 may include a spatialized audio API 106. In some implementations, the spatialized audio API 106 may be a set of computer-executable code that provides functionality to users and / or developers in the form of function calls that enable software components to communicate and / or provide / receive data. The spatialized audio API includes multiple defined software functions related to spatialized audio that can be used by developers to enable spatialized audio functionality in user-created content and may include any functionality related to audio playback on a user device.

[0075] In at least one implementation, the spatialized audio API 106 includes many functions, events, and properties that enable spatialized audio. For example, the spatialized audio API 106 may include functions that include creating and destroying audio channels. These functions may enable the creation of new audio channels associated with a specific server and / or the creation of global audio channels shared among servers in the same metaverse. The functions may also enable the deletion / destruction of previously created audio channels.

[0076] The spatialized audio API 106 may also include functions that include adding and removing players and retrieving players associated with an audio channel. These functions may allow adding a given player or players to a particular audio channel, removing players from a particular audio channel, and / or retrieving a list of players associated with an audio channel in the metaverse. In some implementations, these functions may trigger events, such as when a player joins and / or leaves an audio channel.

[0077] The spatialized audio API 106 may also include functionality that includes creating audio channels that are not associated with a specific player. In this manner, these functions may enable non-player characters, objects, and other virtual items to emit sounds used in the spatialized audio stream. For example, a speaker object may be created that emits sounds as if representing a functioning jukebox with a specific location in the metaverse. Avatars in the vicinity of the speaker object may then receive a spatialized audio stream that includes a transformed audio stream that includes sounds from the speaker object. Sounds produced by non-player characters, objects, and other virtual items may also be incorporated into a background audio stream. This background audio stream may also include sounds produced by some (e.g., one or more) other avatars or player characters.

[0078] The spatialized audio API 106 may also include properties that include parameters or properties related to sound propagation. Thus, the properties may include properties such as the propagation medium (e.g., water, air, etc.), the sound source (e.g., a player or non-player sound source), the volume (e.g., representing the volume of the sound source), the attenuation distance (e.g., the distance at which the sound begins to attenuate), the maximum distance at which the sound can be heard (e.g., if the avatar is further away than this distance, this audio stream will not be included in the spatialized combination), linear or logarithmic sound roll-off (e.g., for a particular roll-off mode), the loudness of the playback, the connection state (e.g., of an audio channel), the mute state (e.g., if the player or sound source is muted), and other properties.

[0079] The spatialized audio API 106 may further include additional functions, variables, properties, and / or parameters that enable rich and immersive spatialized audio to be used in user-created content and / or games. The operation of the online experience platform 102 with respect to utilizing the spatialized audio API 106 to provide spatialized audio (or combined spatialized audio streams) is more fully described hereinafter with reference to Figures 2A and 2B.

[0080] 2A is a diagram of an example network environment 200 (e.g., a subset of network environment 100) for providing spatialized audio chat in a virtual metaverse, according to some implementations. Network environment 200 is provided for illustrative purposes. In some implementations, network environment 200 may include the same, fewer, more, or different elements configured in the same or different manner as shown in FIG. 2A.

[0081] As shown in FIG. 2A, the online experience platform 102 may communicate with the client device 110 (e.g., via a network 122, not shown) such that a user audio stream 232 is received from the client device 110 (e.g., a signal 230 from the system audio input 216) and a combined spatialized audio stream 250 is provided for output at the client device 110 (e.g., via the system audio output 214).

[0082] The online experience platform 102 may include a media server 202 and a data model 206 in addition to the components shown in Figure 1. The client device 110 may include an audio mixer 204, a spatialization audio manager 205, and a sound engine 260 in addition to the components shown in Figure 1.

[0083] Generally, the media server 202 is a purpose-built logical server configured to connect components of the network environment 100 and convey audio streams (or other data) between those components. The media server 202 may, for example, facilitate real-time communication between client devices and the online game server 102 and vice versa.

[0084] The audio mixer 204 may be a software module configured to extract audio streams from multiple players or non-player objects for conversion into a spatialized audio stream. The audio mixer 204 may include an audio mixer override component 208, an echo canceling component 210, and / or an audio device module override component 212.

[0085] The audio mixer override component 208 may be configured to override the underlying audio provided by the online experience platform 102 so that spatialized audio is enabled. For example, the audio mixer override component 208 may receive a copy 251 of the spatialized audio output 250 as well as the user audio stream 234, provide the individual audio streams 238 to the spatialization audio manager 205, and provide the typical audio stream 236 as an output. In this manner, if the audio mixer override component 208 is not initialized, the client device 110 may function to provide normal, non-spatialized audio 239. Similarly, if the audio mixer override component is initialized (e.g., spatialized audio is enabled in the online experience platform 102 or the client device 110 for a particular game 105), the individual audio streams 238 for spatialization transformation are provided to the spatialization audio manager 205.

[0086] The echo canceling component 210 may be configured to cancel echo and / or other undesirable sound artifacts from the audio stream. A filtered output 232 (e.g., echo canceled based on the audio stream 236) may be provided from the echo canceling component 210 to the media server 202. In this manner, the echo canceling component 210 may establish filters or other functionality that assist in providing a high-quality audio stream.

[0087] The audio device module override component 212 may be configured to override and / or disable the system audio output of the client device 110 so that spatialized audio is output instead of standard system audio (e.g., when spatialized audio is not enabled for the online experience platform 102 or the client device 110). The audio device module override component 212 may instead output regular audio 239 when spatialized audio is not enabled.

[0088] The spatialized audio manager 205 may be a software component configured to input one or more user audio streams 238 and convert the streams into spatialized audio streams 242 using the spatialized audio API 106 and associated software functionality. For example, the spatialized audio manager 205 may convert each individual user audio stream 238 into a new individual spatialized audio stream 242. Alternatively, the spatialized audio manager 205 may provide the individual user audio streams 238 to another component for spatialization conversion at the client device 110.

[0089] Additionally, each individual user audio stream 238 may be used to augment physics and / or action commands associated with each avatar. In this manner, each individual user audio stream 238 may be used to implement facial animation synchronized with the audio to create a more realistic and / or immersive experience for the user. For example, when facial animation is synchronized with the spatialized audio, attenuation with distance is achieved while also having visible facial movement that allows the user to identify the avatar producing the sound, thereby further enhancing the user experience. Similarly, each individual audio stream 238 and / or spatialized audio stream 242 may be interpreted to extract emotion and / or intent. In this manner, facial animation may be extracted to further enhance the user experience.

[0090] Additionally, each individual user audio stream 238 may be used for moderation of users on the online experience platform 102. For example, because each individual audio stream is already separated, abusive or foul language may be more easily associated with the associated user. Invocation of a "mute" or remove from audio channel function (through the API 106) may then be used to effectively moderate users associated with abusive behavior. Moderation may be extensible and / or adaptable to machine learning techniques to enable automated moderation tools to analyze vocal behavior, intonation, shouting, and / or utilize natural language processing techniques to identify abusive behavior and automatically moderate associated users.

[0091] The data model 206 may include multiple spatial parameters related to audio transformations. For example, the data model 206 may include one or more spatial parameters that represent a group of physical laws that apply to the metaverse. The physical laws may represent or mimic a real-world sound propagation environment, may represent an exaggerated real-world sound propagation environment, and / or may represent a newly defined sound propagation environment. The sound propagation parameters may be defined through the exposed spatialized audio API 106 by a developer assigning specific values ​​to the parameters. For example, different propagation media, roll-off audio parameters, distance attenuation functions / parameters, and / or reflection parameters may be defined. Similarly, volume parameters, minimum / maximum audibility parameters, and / or propagation parameters may be defined.

[0092] The data model 206 may further include avatar and / or scene information provided by the developer and the actual positioning of the avatar within the metaverse place. The avatar information may include one or more of the avatar's position, velocity, or direction within the metaverse place. The scene information may include one or more of occlusions, reverberations, virtual objects, non-player objects, openings, orifices, reflective surfaces, virtual ceilings, virtual floors, virtual corridors, virtual doorways, and / or virtual walls virtually proximate to the avatar within the metaverse place. The scene information may also include information related to the medium of the surrounding environment (e.g., water, air, etc.). This information / data may be separated based on each avatar within the metaverse place (e.g., as individualized space, avatar, and scene information signals 244) and provided for conversion by the spatialized audio manager 205 and / or respective client devices 110 / 116.

[0093] The multiple spatialized audio streams 246 may then be combined by a sound engine 260 (or alternatively, the spatialized audio manager 205 and / or the game engine 104) into a combined spatialized audio stream 250 for output at the client device 110. The sound engine 260 may include any suitable sound engine, including a sound effects engine and / or a portion of the game engine 104 dedicated to audio effects. In at least one implementation, the sound engine 260 is a proprietary audio effects engine. In other implementations, the sound engine 260 may be a digital effects engine or a game engine.

[0094] An alternative network environment 275 is described hereinafter with reference to FIG. 2B. FIG. 2B is a diagram of an example network environment 275 (e.g., a subset of network environment 100) for providing spatialized audio chat in a virtual metaverse, according to some implementations. Network environment 275 is provided for illustrative purposes. In some implementations, network environment 275 may include the same, fewer, more, or different elements configured in the same or different manner as shown in FIG. 2B.

[0095] 2B , online experience platform 102 may communicate with client device 110 (e.g., via network 122, not shown) such that user audio stream 232 is received from client device 110 (e.g., signal 230 from system audio input 216) and combined spatialized audio stream 250 is provided for output at client device 110 (e.g., via system audio output 214). It is noted that components and / or portions of network environment 275 that are numbered the same as components and / or portions of network environment 200 will not be described repeatedly herein for the sake of brevity.

[0096] An online experience platform 102 similar to the online experience platform 102 shown in FIG. 2A may include a media server 202 and a data model 206. A client device 110 similar to the client device 110 shown in FIG. 2A may include an audio mixer 204 and a spatialization audio manager 205. However, in contrast to the arrangement of FIG. 2A , the spatialization audio manager 205 may provide output 250 according to data received from the game engine 104, individual audio streams 238, and individualized spatial, avatar, and scene information signals 244 (e.g., combined signal 276). In this manner, the spatialization audio manager 205 may include the functionality of a sound engine embedded or implemented therein. Alternatively, a standalone sound engine component or components may be used to provide the spatialized audio output 250.

[0097] In this manner, the spatialized audio manager 205 provides a combined spatialized audio stream 250 based on data received from the game engine 104 and the data model 206. For example, the spatialized audio manager 205 may receive each individual user audio stream 238 as a new individual spatialized audio stream 276 based on the spatial, avatar, and scene information signals 244 (e.g., combined at an audio sink).

[0098] As described above with reference to FIGS. 2A and 2B , multiple spatialized audio streams 246 / 276 (or individual non-spatialized audio streams 238) may be combined to generate a combined spatialized audio stream 250 for output at a particular client device 110. Given the potentially large number of audio streams available in any particular virtual metaverse or virtual environment, some implementations provide audio stream prioritization. The audio stream prioritization (sometimes referred to as “subscription” to audio streams) allows a reduced set of prioritized streams (e.g., compared to all available streams) to be converted into the spatialized audio stream. This reduced set of streams provides technical advantages and benefits, including reduced computational cycles for generating the combined spatialized audio stream, reduced system resource usage, energy savings, and reduced bandwidth usage.

[0099] Prioritization of spatialized audio streams utilizing the spatialized audio API 106 and available avatar and scene data is more fully described below with reference to FIG.

[0100] 3 is a diagram of an example network environment 300 (e.g., a subset of network environment 100) for wired ranking of spatialized audio streams in a virtual metaverse, according to some implementations. Network environment 300 is provided for illustrative purposes. In some implementations, network environment 300 may include the same, fewer, more, or different elements configured in the same or different manner as shown in FIG. 3.

[0101] 3, media server 202 and client device 110 may communicate (e.g., via network 122, not shown) such that published audio streams 330 are received from client device 110 and a set of prioritized audio streams 350 are provided for conversion, combination, and output at client device 110. It is noted that in some implementations, prioritized audio streams 350 may be converted and combined at online experience platform 102 rather than at client device 110. All such modifications are within the scope of this disclosure.

[0102] 1 and 2A-2B, the client device 110 may include a real-time communication module 302 and a subscription logic 312. The media server 202 may include a real-time communication module 306, a subscription exchange component 304, and a subscription prioritizer 310, in addition to the components shown in FIG.

[0103] The real-time communication modules 302 / 306 may be software components or instances of a real-time communication server instantiated on the client device 110 and the media server 202, respectively. In at least one implementation, the real-time communication modules 302 / 306 are instances of a WebRTC server implemented with an exposed WebRTC API (not shown). The real-time communication modules 302 / 306 are configured to pass exposed streams 330 from each connected client device and a set of prioritized audio streams 350 to each connected client device.

[0104] The subscription exchange component 304 is a software component configured to receive estimates 332 / 333 of at least a portion of head-related transfer functions (HRTFs) and / or Baum-Welch (BW) hidden Markov models from the real-time communication modules 306 / 302 and prioritize a set of audio streams into a set 331 / 351 for output to a client device.

[0105] The subscription prioritizer 310 retrieves appropriate and / or relevant spatial parameters 336 from the data model 206 and retrieves a number of subscription requests 334 from the spatialized audio API 106. Using the spatial parameters and subscription requests, the subscription prioritizer prioritizes the available audio streams into a prioritized set 350. The prioritization may be based on a number of factors, including, for example, the proximity of the avatar within the metaverse place, the speed of the avatar within the metaverse place, the direction of the avatar within the metaverse place, virtual objects in proximity to the avatar within the metaverse place, the capabilities of the user device, bandwidth availability, the number of connections to the media server 202, the total number of users using spatialized audio, and / or user preferences of users associated with the client device 110. Further factors may include available bandwidth (e.g., between the media server 202 and the client device 110), available processing power (e.g., in the media server 202, the client device 110, and / or a combination), available memory (e.g., in the media server 202, the client device 110, and / or a combination), and other factors.

[0106] The subscription requests are based on subscription logic 312 configured to issue individual subscription requests 338 based on spatial parameters 340 for a particular avatar of a particular user. In this manner, each individual client device issues a different subscription request 338 based on its associated avatar and spatial parameters 340 and other factors 342, which may include, for example, HRTF parameters, BW Hidden Markov Model parameters, and / or VAD parameters.

[0107] As discussed above, the prioritized set of streams may be based on a number of spatial parameters related to avatars, items, objects, and other features of the metaverse, as well as computational, storage, bandwidth, and other resources. Examples of aspects related to spatialized audio are provided hereinafter with reference to Figures 4 and 5.

[0108] Figures 4 and 5: Examples of spatialized audio in the metaverse. 4 and 5 illustrate exemplary virtual environments, such as metaverse place 400 and metaverse place 500, within an online virtual experience, according to some implementations. Metaverse place 400 includes a first avatar 402, a second avatar 404, and a third avatar 406. The avatars (402-406) may include avatars controlled by their respective users and / or avatars (e.g., computer-generated characters) under the automated control of online experience platform 102.

[0109] In addition to the avatars 402-406, the metaverse place 400 includes a first virtual object 408, a second virtual object 410, a third virtual object 412, and a fourth virtual object 422. The virtual objects may represent, among other things, buildings, building components (e.g., walls, windows, doors, etc.), bodies of water (e.g., ponds, rivers, lakes, oceans, etc.), furniture, machinery, vehicles, plants, animals, etc. The virtual objects (408-412) may include associated data or metadata corresponding to one or more object characteristics, such as material type (e.g., metal, wood, cloth, stone, etc.), the object's location within the metaverse place 400, the object's size, the object's shape, or the object's sound characteristics (e.g., the ambient sound the object emits, how frequently the sound is emitted, the volume of the sound, etc.).

[0110] The sound characteristics of an object may be based on object type, object size, object shape, or object location. For example, an object representing a large adult dog may have sound characteristics typical of an adult dog (object type) that is large (object size) and at a given location relative to the character (object location). The sound characteristics may include ambient sounds (e.g., barks) emitted by the object, which may be provided by a sound file (e.g., computer-generated or recorded sounds). Additionally, the sound characteristics may include the frequency with which the object makes a sound (e.g., how often the dog barks) and the volume of the ambient sounds (e.g., how loud the dog barks at a given distance). The volume of the object's ambient sounds may then be modified as part of the audio spatialization process (e.g., dog barks may be made louder for dogs closer to the character and quieter for dogs further away from the character).

[0111] 4, the objects may include furniture or parts of a building. For example, virtual object 408 may be a doorway or a virtual boundary between metaverse places. Virtual object 410 and virtual object 412 may be walls. Virtual object 422 may be a small table or other virtual furniture.

[0112] Avatar 402 may be speaking and emitting simulated sounds, as shown by sound paths 414 and 416. The voice of avatar 402 along sound path 414 comes from the left of avatar 406, while sound path 416 comes from the right of avatar 404 through virtual doorway 408. The sound path between avatar 402 and avatar 406 is generally direct, while the sound path from avatar 402 to avatar 404 is partially reflected off walls 410 and 412, as shown by sound path 416.

[0113] Additionally, ambient sounds may be emitted by table 422. For example, the table may include speakers or other objects emitting sounds on the table. Ambient sounds are indicated by sound paths 418 and 419. In some implementations, ambient sounds may include sounds of wind, rain, music, machinery, animals, movement of items, footsteps, etc. Ambient sounds may also include sounds emitted from objects (e.g., cars) that may be stationary or moving within the environment. Ambient sounds may also include sounds produced by other avatars moving through or interacting with environment 400 and / or objects within environment 400.

[0114] In operation, implementations of the audio spatialization techniques described herein may perform one or more of the following operations based on the exemplary metaverse shown in FIG. 4: 1) spatialization of the voice communication of an avatar (e.g., avatar 402) based on the position of each receiving avatar (e.g., 404 or 406) relative to the speaking avatar; 2) audio spatialization of the voice communication based on any object (e.g., 408, 410, 412, or 422) in the virtual environment; and 3) audio spatialization of environmental sounds (e.g., sounds emitted by object 410) in the virtual environment.

[0115] 5, there is shown a top-down view of a metaverse place 500. As shown, a simplified schematic diagram of avatars 502, 504, 506, and 508 includes a distance radius 525 measured with respect to avatar 502. In this example, avatar 502 may represent a user requesting spatialized audio, and distance radius 525 may be a parameter or setting used to prioritize audio streams.

[0116] As further shown, avatar position and velocity data 541, 561, and 581 are represented by arrows emanating from the respective avatars. In this example, a user device associated with avatar 502 may receive prioritized audio streams associated with avatars 504 and 506, as well as ambient sounds associated with any objects or non-player items within radius 525. However, if avatar 508 continues to approach radius 525 as indicated by arrow 581, data associated with avatar 508 may also be arranged to be prioritized. Alternatively, if computational resources or other parameters permit, data associated with avatar 508 may also be included in the prioritized audio streams until resources or other prioritization parameters change (e.g., additional avatars approach avatar 502, computational resource usage increases above a threshold, additional audio streams such as background audio are prioritized, etc.).

[0117] In this way, a subset of the available audio streams is prioritized such that computing resources are reduced while still effectively delivering rich, immersive audio to the client device.

[0118] Hereinafter, a more detailed discussion of spatialized audio stream creation and audio stream prioritization will be provided with reference to FIGS.

[0119] FIG. 6: An exemplary method for creating a spatialized audio stream. 6 is a flow diagram of an example method 600 for creating spatialized audio in a metaverse place, according to some implementations. In some implementations, method 600 may be implemented, for example, on a server system, such as the online experience platform 102 shown in FIG. 1. In some implementations, some or all of method 600 may be implemented on a system, such as one or more client devices 110 and 116 shown in FIG. 1, and / or on both the server system and one or more client systems. In the illustrated example, the implementing system includes one or more processors or processing circuits and one or more storage devices, such as a database or other accessible storage. In some implementations, different components of one or more servers and / or clients may perform different blocks or other portions of method 600. Method 600 may begin at block 602.

[0120] At block 602, a request to receive audio related to a metaverse place in the virtual metaverse may be received (e.g., from a first user of multiple users). For example, client device 110 (also referred to as a user device) may be associated with the first user. The first user is associated with a first avatar. Furthermore, multiple users may be associated with multiple avatars in the metaverse place (e.g., other avatars involved in the metaverse place). Block 602 is followed by block 604.

[0121] In block 604, a data model associated with the metaverse place is retrieved. For example, the data model 206 may be stored in the data store 108. The data model may include one or more spatial parameters that represent a group (e.g., one or more) of physical laws that apply to the metaverse place. These physical laws may be exaggerated (compared to physical laws that may apply on Earth) to enhance the spatial effect, or may be attenuated to implement only a slight spatial effect. The parameters of sound propagation and the underlying physical laws may be adjusted and / or modified through the spatialized audio API 106 described above. Block 604 is followed by block 606.

[0122] At block 606, avatar information and scene information are extracted from the data model in response to the request. For example, the request may be associated with a particular client device and, therefore, a particular avatar. In this manner, the extracted avatar information includes one or more of the position, velocity, or direction of the particular avatar and multiple avatars proximate to the particular avatar in the metaverse. Similarly, the scene information includes one or more of occlusions, reverberations, virtual objects, non-player objects, openings, orifices, reflective surfaces, virtual ceilings, virtual floors, and / or virtual walls virtually proximate to the particular avatar. Block 606 is followed by block 608.

[0123] In block 608, each audio stream received from each of the multiple users is transformed using the extracted spatial parameters. The transformation is based on avatar information and scene information. The transformation may include modifying one or more audio characteristics to create a spatialized audio stream. For example, attenuation based on a distance attenuation parameter for attenuating audio based on the distance between avatars defined in the data model may be used to modify the audio characteristics. Similarly, audio rolling in or rolling out (e.g., using a "fading" or "rolling" effect) may be performed to modify the audio characteristics. Additionally, volume increase, volume decrease, Doppler shift, reverberation, or reflection may be provided, and / or other characteristics may be modified. In this manner, the transformation outputs a spatialized audio stream for each individual avatar and / or virtual object / item in the metaverse. Block 608 is followed by block 610.

[0124] At block 610, the spatialized audio streams are combined to create a combined spatialized audio stream. The combination may be performed at the online experience platform 102, at the client device 110 / 116, and / or by a combination of the online experience platform and the client device. In some implementations, the combination may be performed with respect to only a prioritized set of audio streams. In other implementations, a subset of the available audio streams is transformed based on proximity or a threshold distance to an avatar in the metaverse. Other variations and limitations on the number of audio streams transformed are possible and are within the scope of this disclosure.

[0125] According to at least one implementation, a stream of background audio is also combined with the spatialized audio stream to create a combined spatialized audio stream. For example, a background or “special” stream that mixes multiple participants into background noise / chat may provide a more realistic experience. For example, in a room with 50 people talking, an avatar may be conversing with several avatars in its immediate vicinity. While the audio from the closest participants may be the clearest, there may also be background chatter around the avatar (e.g., pure silence would not feel realistic). Therefore, the background stream may be pre-mixed to include the overall background chatter from the remaining 50 avatars in a relatively simple manner (e.g., a uniform background stream from all participants used by the combined stream to all participants). In this way, the background audio may be generated based on one or more of audio received from other users different from the first user, audio generated based on the avatar's movements within the metaverse, and / or an overall or “special” background stream. Block 610 is followed by block 612.

[0126] At block 612, the combined spatialized audio stream is provided to the user device for output through an audio output device connected to the user device, for example, a set of speakers or headphones. In some implementations, the spatialized audio stream may be provided through an audio output device such as a virtual reality headset, an augmented reality headset, a head-mounted device, or the like.

[0127] Blocks 602-612 may be performed (or repeated) in a different order than described above, and / or one or more blocks may be omitted. For example, data extraction (blocks 604-606) may be performed independently of audio conversion and combination (blocks 608-612). Furthermore, receiving a request, extracting relevant data, and converting audio may be performed in parallel or by different components in some implementations.

[0128] A more detailed discussion of stream prioritization is provided below with reference to FIG.

[0129] FIG. 7: An exemplary method for prioritizing spatialized audio streams 7 is a flow diagram of an example method 700 for providing content to a user based on classification, according to some implementations. In some implementations, method 700 may be implemented, for example, on a server system, such as the online experience platform 102 shown in FIG. 1. In some implementations, some or all of method 700 may be implemented on a system such as one or more client devices 110 and 116 shown in FIG. 1, and / or on both the server system and one or more client systems. In the illustrated example, the implementing system includes one or more processors or processing circuits and one or more storage devices, such as a database or other accessible storage. In some implementations, different components of one or more servers and / or clients may perform different blocks or other portions of method 700. Method 700 may begin at block 702.

[0130] At block 702, a request to receive audio related to a metaverse place of a virtual metaverse may be received (e.g., from a first user of a plurality of users). For example, client device 110 may be associated with the first user. The first user is associated with a first avatar. Furthermore, multiple users may be associated with multiple avatars in the metaverse place (e.g., avatars participating in the metaverse place). Block 702 is followed by block 704.

[0131] At block 704, a prioritized set of audio streams is determined for the first user. The prioritized set of audio streams is a ranked subset of all audio streams received from each user of the plurality of users in the metaverse and all other audio streams (e.g., ambient sounds, sounds of non-player characters / items, etc.). The prioritization may be based, for example, on a threshold distance and / or a threshold radius (e.g., as shown in FIG. 5).

[0132] The prioritization may also be based on the proximity of the avatar within the metaverse place, the speed of the avatar within the metaverse place, the direction of the avatar within the metaverse place, virtual objects in proximity to the avatar within the metaverse place, the capabilities of the user device, or user preferences of the first user. For example, the prioritization may take into account the avatar moving toward a target avatar, the avatar facing toward a target avatar, and other similar prioritization parameters.

[0133] Prioritization may also be based on the processing resources and / or capabilities of the client device. For example, prioritization may take into account memory usage, disk usage, bandwidth availability, processor usage, and other resource usage to determine whether there are sufficient resources to handle a certain number of spatialized audio streams. Prioritization may then prioritize the streams such that a certain number of streams are prioritized to avoid contention or further resource utilization beyond a threshold.

[0134] Prioritization may also be based on the processing resources and / or capabilities of the media server. For example, prioritization may take into account memory usage, storage usage, bandwidth availability, processor usage, number of active connections, number of inactive connections, total number of users, and other resource usage to determine whether there are sufficient resources to handle a certain number of spatialized audio streams. Prioritization may then prioritize the streams such that a certain number of streams are prioritized to avoid contention or further resource utilization beyond a threshold.

[0135] Prioritization may also be based on the processing resources and / or capabilities of the online experience platform. For example, prioritization may take into account memory usage, storage usage, bandwidth availability, processor usage, number of active connections, number of inactive connections, total number of users, number of active online experiences, and other resource usage to determine whether there are sufficient resources to handle a certain number of spatialized audio streams. Prioritization may then prioritize the streams such that a certain number of streams are prioritized to avoid contention or further resource utilization beyond a threshold.

[0136] Other variations on the basis for prioritization are possible and within the scope of this disclosure. Block 704 is followed by block 706.

[0137] In block 706, each audio stream of the prioritized audio streams is transformed using the extracted spatial parameters. The transformation is based on avatar information and scene information. The transformation may include modifying one or more audio characteristics to create a spatialized audio stream. For example, attenuation based on a distance attenuation parameter for attenuating audio based on the distance between avatars defined in the data model may be used to modify the audio characteristics. Similarly, audio roll-in or roll-out may be performed to modify the audio characteristics. Additionally, volume rise, volume fall, Doppler shift, reverberation, reflections, and other characteristics may be modified. In this manner, the transformation outputs a spatialized audio stream for each individual avatar and / or virtual object / item associated with the prioritized audio stream. Block 706 is followed by block 708.

[0138] At block 708, the spatialized audio streams are combined to create a combined spatialized audio stream. The combination may be performed at the online experience platform 102, at the client device 110 / 116, and / or by a combination of the online experience platform and the client device. As described above, special or background audio streams may also be combined to provide ambient and / or background audio for the spatialized audio output. The combination may be performed with respect to only a prioritized set of audio streams. Block 708 is followed by block 710.

[0139] At block 710, the combined spatialized audio stream is provided to the user device for output through an audio output device connected to the user device, for example, a set of speakers or headphones.

[0140] Blocks 702-710 may be performed (or repeated) in a different order than described above, and / or one or more blocks may be omitted. Methods 600 and / or 700 may be performed on a server (e.g., 102) and / or a client device (e.g., 110 or 116). Furthermore, portions of methods 600 and 700 may be combined and performed sequentially or in parallel according to any desired implementation.

[0141] As described above, the systems, methods, and computer-readable media may provide spatialized audio in a virtual experience. By providing a robust spatialized audio API, developers have the potential to use spatialized audio in nearly every virtual experience they create. The immersive qualities of a typical virtual experience augmented with a spatialized audio API provide a rich user experience, increase user engagement, provide intuitive feedback (e.g., through audio location), and significantly reduce the complexity of implementing spatialized audio.

[0142] Hereafter, a more detailed description of various computing devices that may be used to implement the different devices shown in FIGS. 1-3 will be provided with reference to FIG.

[0143] 8 is a block diagram of an exemplary computing device 800 that may be used to implement one or more features described herein, according to some implementations. In one example, device 800 implements a computer device (e.g., 102, 110, and / or 116 of FIG. 1 ) and may be used to perform implementations of suitable methods described herein. Computing device 800 can be any suitable computer system, server, or other electronic or hardware device. For example, computing device 800 can be a mainframe computer, desktop computer, workstation, portable computer, or electronic device (portable device, mobile device, cell phone, smartphone, tablet computer, television, TV set-top box, personal digital assistant (PDA), media player, gaming device, wearable device, etc.). In some implementations, the device 800 includes a processor 802, memory 804, an input / output (I / O) interface 806, and audio / video input / output devices 814 (e.g., a display screen, a touch screen, display goggles or glasses, audio speakers, headphones, a microphone, etc.).

[0144] Processor 802 can be one or more processors and / or processing circuits for executing program code and controlling the basic operations of device 800. A "processor" includes any suitable hardware and / or software system, mechanism, or component for processing data, signals, or other information. A processor may include a general-purpose central processing unit (CPU), multiple processing units, a system with dedicated circuitry for implementing functions, or other systems. Processing need not be limited to a particular geographic location or have temporal limitations. For example, a processor may perform its functions in "real time," "offline," "batch mode," etc. Portions of processing may be performed at different times and in different locations by different (or the same) processing systems. A computer may be any processor in communication with a memory.

[0145] Memory 804 is generally provided within device 800 for access by processor 802, is suitable for storing instructions for execution by the processor, and may be any suitable processor-readable storage medium, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc., located separately from and / or integrated with processor 802. Memory 804 may store software operated by processor 802 on server device 800, including operating system 808, applications 810, and associated data 812. In some implementations, applications 810 may include instructions that enable processor 802 to perform some or all of the functions described herein, e.g., the methods of FIGS.

[0146] For example, memory 804 may include software instructions for prioritizing and / or providing spatialized audio within an online experience platform (e.g., 102) or metaverse. Any of the software in memory 804 may alternatively be stored in any other suitable storage location or computer-readable medium. Additionally, memory 804 (and / or other connected storage devices) may store instructions and data used in the features described herein. Memory 804 and any other type of storage (such as magnetic disks, optical disks, magnetic tape, or other tangible media) may be considered “storage” or “storage devices.”

[0147] The I / O interface 806 may provide functionality to allow the server device 800 to interface with other systems and devices. For example, network communication devices, storage devices (e.g., memory and / or data store 108), and input / output devices may communicate through the interface 806. In some implementations, the I / O interface may connect to interface devices including input devices (keyboards, pointing devices, touchscreens, microphones, cameras, scanners, etc.) and / or output devices (display devices, speaker devices, printers, monitors, etc.).

[0148] For ease of illustration, FIG. 8 shows one block for each of processor 802, memory 804, I / O interface 806, software blocks 808 and 810, and database 812. These blocks may represent one or more processors or processing circuits, operating systems, memories, I / O interfaces, applications, and / or software modules. In other implementations, device 800 may have less than all of the components shown and / or may have other elements, including other types of elements instead of or in addition to the elements shown herein. Although online experience platform 102 is described as performing the operations as described in some implementations herein, any suitable component or combination of components of online experience platform 102 or a similar system, or any suitable processor or processors associated with such a system, may perform the described operations.

[0149] User devices may also implement and / or be used with the features described herein. An exemplary user device may be a computing device including several components similar to device 800, e.g., processor 802, memory 804, and I / O interface 806. An operating system, software, and applications suitable for a client device may be provided in the memory and used by the processor. The I / O interface for the client device may be connected to a network communication device and input and output devices, e.g., a microphone for capturing sound, a camera for capturing images or video, an audio speaker device for outputting sound, a display device for outputting images or video, or other output devices. For example, a display device within audio / video input / output device(s) 814 may be connected to (or included in) device 800 to display pre- and post-processing images as described herein; such a display device may include any suitable display device, e.g., an LCD, LED, or plasma display screen, a CRT, a television, a monitor, a touchscreen, a 3-D display screen, a projector, or other visual display device. Some implementations may provide an audio output device, for example, text-to-speech voice output or synthesis.

[0150] Methods, blocks, and / or operations described herein may, where appropriate, be performed in a different order than shown or described and / or performed concurrently (partially or completely) with other blocks or operations. Some blocks or operations may be performed for one portion of data and, for example, performed again later for another portion of data. Not all of the described blocks and operations may be performed in various implementations. In some implementations, blocks and operations may be performed multiple times, in different orders, and / or at different times within a method.

[0151] In some implementations, some or all of the methods may be implemented on a system, such as one or more client devices. In some implementations, one or more methods described herein may be implemented, for example, on a server system and / or on both a server system and a client system. In some implementations, different components of one or more servers and / or clients may perform different blocks, operations, or other portions of a method.

[0152] One or more methods described herein (e.g., methods 600 and / or 700) can be implemented by computer program instructions or code that can be executed on a computer. For example, the code can be executed by one or more digital processors (e.g., microprocessors or other processing circuits) and stored in a computer program product that includes a non-transitory computer-readable medium (e.g., storage medium), such as a magnetic, optical, electromagnetic, or semiconductor storage medium, including semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), flash memory, rigid magnetic disks, optical disks, solid-state memory drives, etc. The program instructions can also be contained in and provided as electronic signals, for example, in the form of software as a service (SaaS) delivered from a server (e.g., a distributed system and / or a cloud computing system). Alternatively, one or more methods can be implemented in hardware (such as logic gates) or a combination of hardware and software. Exemplary hardware can be a programmable processor (e.g., a field programmable gate array (FPGA), complex programmable logic device), general-purpose processor, graphics processor, application-specific integrated circuit (ASIC), etc. One or more methods may be implemented as part of or a component of an application running on the system, or as an application or software running in conjunction with other applications and the operating system.

[0153] One or more methods described herein may be performed within a standalone program that may run on any type of computing device, a program running on a web browser, a mobile application ("app") running on a mobile computing device (e.g., a cell phone, a smartphone, a tablet computer, a wearable device (such as a watch, armband, jewelry, hat, goggles, glasses, etc.), a laptop computer, etc.). In one example, a client / server architecture may be used, e.g., a mobile computing device (as a client device) sends user input data to a server device and receives final output data for output (e.g., for display) from the server. In another example, all computations may be performed within a mobile app (and / or other apps) on the mobile computing device. In another example, computations may be split between the mobile computing device and one or more server devices.

[0154] Although the description has been given in connection with specific implementations thereof, these specific implementations are illustrative only and not limiting, and the concepts illustrated in the examples may be applied to other examples and implementations.

[0155] In situations where certain implementations discussed herein may obtain or use user data (e.g., user demographics, user behavioral data on the platform, user search history, items purchased and / or viewed, user relationships on the platform, etc.), users are provided with choices to control whether and how such information is collected, stored, or used. That is, implementations discussed herein collect, store, and / or use user information with explicit user authorization and in compliance with any applicable regulations.

[0156] A user is allowed to control whether a program or feature collects user information about that particular user or other users associated with the program or feature. Each user about whom information is to be collected is presented with options (e.g., via a user interface) that allow the user to exercise control over information collection associated with that user to grant permission or authorization regarding whether information is to be collected and what portions of the information should be collected. Additionally, certain data may be modified in one or more ways before being stored or used so that personally identifiable information is removed. As one example, a user's identity may be modified (e.g., by substitution using a pseudonym, numeric values, etc.) so that personally identifiable information cannot be determined. As another example, a user's geographic location may be generalized to a larger region (e.g., city, zip code, state, country, etc.).

[0157] It should be noted that the functional blocks, operations, features, methods, devices, and systems described in this disclosure may be integrated or separated into different combinations of systems, devices, and functional blocks as known to those skilled in the art. Any suitable programming language and programming techniques may be used to implement the routines of a particular implementation. Different programming techniques, for example, procedural or object-oriented programming techniques, may be used. The routines may be executed on a single processing device or multiple processors. While steps, operations, or computations may be presented in a particular order, the order may be changed in different particular implementations. In some implementations, multiple steps or operations shown herein as sequential may be performed simultaneously. [Explanation of symbols]

[0158] 100 Network Environment 102 Online Experience Platform 104 Game Engine, Virtual Experience Engine 105 Games, Virtual Experiences, and Gaming Applications 106 Spatial Audio API, Spatialized Audio API 108 Datastores 110 / 116 client devices 110 first client device 112 Game applications, virtual experience applications 114 users 116 Second Client Device 118 Game applications, virtual experience applications 120 users 122 Network 200 Network Environment 202 Media Server 204 Audio Mixer 205 Spatialized Audio Manager 206 Data Model 208 Audio Mixer Override Component 210 Echo Cancellation Components 212 Audio Device Module Override Component 214 System Audio Output 216 System Audio Inputs 230 signal 232 User Audio Stream, Filtered Output 234 User Audio Stream 236 Typical Audio Stream 238 individual audio streams 239 Normal non-spatialized audio, non-spatialized audio stream 242 spatialized audio stream 244 Spatial, Avatar, and Scene Information Signals 246 spatialized audio stream 250 combined spatialized audio streams, spatialized audio output 251 Copy spatialized audio output 260 Sound Engine 275 Alternative Network Environments 276 Combined Signals 300 Network Environment 302 Real-time Communication Module 304 Subscription Exchange Components 306 Real-time Communication Module 310 Subscription Prioritizer 312 Subscription Logic 330 public audio streams 331 sets 332 Estimates of at least part of the head-related transfer functions (HRTFs) and / or Baum-Welch (BW) hidden Markov models 333 Head-Related Transfer Functions (HRTFs) and / or Baum-Welch (BW) Hidden Markov Model estimates of at least some of the 334 Subscription Request 336 Spatial Parameters 338 Subscription Request 340 Spatial Parameters 342 Other Factors 350 prioritized audio streams, prioritized sets 351 sets 400 Metaverse Places 402 First Avatar 404 Second Avatar 406 The Third Avatar 408 First Virtual Object, Virtual Doorway 410 Second Virtual Object, Wall 412 The third virtual object, the wall 414 Sound Path 416 Sound Path 418 Sound Path 419 Sound Path 422 Fourth Virtual Object: Table 500 Metaverse Places 502 Avatar 504 Avatar 506 Avatar 508 Avatar 525 distance radius 541 avatar position and velocity data 561 avatar position and velocity data 581 Avatar position and velocity data 600 ways 700 methods 800 computing devices 802 processor 804 memory 806 Input / Output (I / O) Interface 808 Operating System 810 Application 812 Related Data, Databases 814 Audio / Video Input / Output Devices

Claims

1. 1. A computer-implemented method for spatialized audio in a virtual metaverse, comprising: receiving a request to receive audio associated with a metaverse place of the virtual metaverse from a first user of a plurality of users, the first user being associated with a user device, and the plurality of users being associated with respective avatars of a plurality of avatars in the metaverse place; retrieving a data model associated with the metaverse place, the data model including one or more spatial parameters representing one or more physical laws that apply to the metaverse place; extracting avatar information and scene information from the data model, wherein the avatar information includes one or more of position, velocity, or orientation of the plurality of avatars in the metaverse place including a first avatar associated with the first user, and the scene information includes one or more of occlusion, reverberation, or virtual walls virtually proximate to the first avatar in the metaverse place; transforming a respective audio stream received from each of the plurality of users based on the avatar information and the scene information to create a spatialized audio stream, and transforming one or more audio characteristics of at least one of the respective audio streams based on the one or more spatial parameters; combining the spatialized audio streams to create a combined spatialized audio stream; providing the combined spatialized audio stream to the user device; A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein the spatial parameters include a distance attenuation parameter for attenuating audio based on distance between avatars.

3. 2. The computer-implemented method of claim 1, wherein the respective audio streams received from each of the plurality of users include mono audio received at a microphone device, and the combined spatialized audio stream includes stereo audio.

4. 4. The computer-implemented method of claim 3, wherein the combined spatialized audio stream includes stereo audio generated by placing each user's mono audio at the location of their respective user's avatar.

5. the combined spatialized audio stream includes spatial audio and background audio based on the audio streams received from users of the plurality of users other than the first user, the background audio comprising: audio received from other users different from the first user; and Audio generated based on the movements of an avatar within the metaverse.

10. The computer-implemented method of claim 1, wherein the method is generated based on one or more of:

6. 2. The computer-implemented method of claim 1, further comprising: determining a set of prioritized audio streams received from each user of the plurality of users; and wherein transforming each audio stream further comprises transforming the set of prioritized audio streams to create the spatialized audio stream.

7. determining the set of prioritized audio streams comprises:

7. The computer-implemented method of claim 6, comprising prioritizing the audio streams received from each user of the plurality of users based on one or more of: proximity of an avatar in the metaverse place, speed of an avatar in the metaverse place, direction of an avatar in the metaverse place, virtual objects in proximity to an avatar in the metaverse place, capabilities of the user device, or user preferences of the first user.

8. 8. The computer-implemented method of claim 7, wherein audio streams associated with avatars closer to the receiving avatar are prioritized over audio streams associated with avatars farther from the receiving avatar, audio streams associated with avatars pointed towards the receiving avatar are prioritized over audio streams associated with avatars pointed away from the receiving avatar, and audio streams associated with avatars moving towards the receiving avatar are prioritized over audio streams associated with avatars moving away from the receiving avatar.

9. 1. A computer-implemented method for providing spatialized audio in a virtual metaverse, comprising: receiving a request from a first user of a plurality of users to receive audio associated with a metaverse place of the virtual metaverse, the first user being associated with a user device and the plurality of users being associated with respective avatars of a plurality of avatars in the metaverse place; determining a set of prioritized audio streams received from each user of the plurality of users; transforming the set of prioritized audio streams to create a spatialized audio stream; combining the spatialized audio streams to create a combined spatialized audio stream; providing the combined spatialized audio stream to the user device; A computer-implemented method comprising:

10. determining the set of prioritized audio streams comprises:

10. The computer-implemented method of claim 9, comprising prioritizing the audio streams received from each user of the plurality of users based on one or more of: proximity of an avatar in the metaverse place, speed of an avatar in the metaverse place, direction of an avatar in the metaverse place, virtual objects in proximity to an avatar in the metaverse place, capabilities of the user device, or user preferences of the first user.

11. 11. The computer-implemented method of claim 10, wherein audio streams associated with avatars closer to the receiving avatar are prioritized over audio streams associated with avatars farther from the receiving avatar, audio streams associated with avatars pointed towards the receiving avatar are prioritized over audio streams associated with avatars pointed away from the receiving avatar, and audio streams associated with avatars moving towards the receiving avatar are prioritized over audio streams associated with avatars moving away from the receiving avatar.

12. 1. A system comprising: a memory in which instructions are stored; a processing device coupled to the memory and configured to access the memory, wherein the instructions, when executed by the processing device, cause the processing device to: receiving a request from a first user of a plurality of users to receive audio associated with a metaverse place of a virtual metaverse, the first user being associated with a user device and the plurality of users being associated with respective avatars of a plurality of avatars in the metaverse place; retrieving a data model associated with the metaverse place, the data model including one or more spatial parameters representing one or more physical laws that apply to the metaverse place; extracting avatar information and scene information from the data model, wherein the avatar information includes one or more of position, velocity, or orientation of the plurality of avatars in the metaverse place including a first avatar associated with the first user, and the scene information includes one or more of occlusion, reverberation, or virtual walls virtually proximate to the first avatar in the metaverse place; transforming a respective audio stream received from each of the plurality of users based on the avatar information and the scene information to create a spatialized audio stream, and transforming one or more audio characteristics of at least one of the respective audio streams based on the one or more spatial parameters; combining the spatialized audio streams to create a combined spatialized audio stream; providing the combined spatialized audio stream to the user device; A system that causes an operation including

13. The system of claim 12 , wherein the spatial parameters include a distance attenuation parameter for attenuating audio based on distance between avatars.

14. 13. The system of claim 12, wherein the respective audio streams received from each of the plurality of users include mono audio received at a microphone device, and the combined spatialized audio stream includes stereo audio.

15. 15. The system of claim 14, wherein the combined spatialized audio stream includes stereo audio generated by placing each user's mono audio at the location of the respective user's avatar.

16. the combined spatialized audio stream includes spatial audio based on the audio streams received from users of the plurality of users other than the first user, and background audio, the background audio comprising: audio received from other users different from the first user; and Audio generated based on the movements of an avatar within the metaverse. The system of claim 12, wherein the information is generated based on one or more of:

17. The operation is 13. The system of claim 12, further comprising: determining a set of prioritized audio streams received from each user of the plurality of users; and transforming each audio stream further comprising transforming the set of prioritized audio streams to create the spatialized audio stream.

18. determining the set of prioritized audio streams, 18. The system of claim 17, further comprising prioritizing the audio streams received from each user of the plurality of users based on one or more of: proximity of an avatar in the metaverse place, speed of an avatar in the metaverse place, direction of an avatar in the metaverse place, virtual objects in proximity to an avatar in the metaverse place, capabilities of the user device, or user preferences of the first user.

19. 20. The system of claim 18, wherein audio streams associated with avatars closer to the receiving avatar are prioritized over audio streams associated with avatars further away from the receiving avatar, and audio streams associated with avatars directed towards the receiving avatar are prioritized over associated audio streams.

20. a spatialization audio manager configured to transform the respective audio streams received from each user of the plurality of users; an audio device override module configured to disable non-spatialized audio at the user device before providing the combined spatialized audio stream to the user device; The system of claim 12 further comprising:

Images