Spatialized audio chat in virtual meta-universe

Abstract

Implementations described herein relate to methods, systems, and computer-readable media that provide spatialized audio in a virtual experience. Spatialized audio can be used for voice communications, such as voice / video chats. Chats can include spatialized audio that is combined on a client device or an online experience platform and targeted to a particular user. A single audio stream can be collected from multiple avatars and other objects and combined based on a target user. Audio can also include background and / or ambient sounds to provide a rich, immersive audio stream in a virtual experience.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 222,304, filed July 15, 2021, entitled “Spatialized Audio Chat in a Virtual Metaverse,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The embodiments generally relate to audio output via computer devices, and more particularly to methods, systems, and computer-readable media for providing spatialized audio in virtual immersive environments (e.g., metaverse locations of virtual metaverses). Background Technology

[0004] Computer audio (e.g., chat between users of a computer device) often consists of mono or stereo audio received from a listening device or microphone. While the two avatars represent actual virtual locations that signify the users participating in the chat, the provided audio is typically unfiltered or poorly filtered and may sound dull or simplistic. Therefore, as virtual experiences become more visually immersive, the simplicity of the provided audio can be distracting and / or detrimental to the immersive experience, for example, causing users to detach from the experience.

[0005] The background description provided herein is intended to introduce the background of this disclosure. Within the scope described in this background section and in aspects that may not conform to the prior art at the time of application, the work of the currently designated inventor is neither explicitly nor implicitly acknowledged as prior art to this disclosure. Summary of the Invention

[0006] Various embodiments of this application relate to providing spatialized audio in a virtual metaverse.

[0007] According to one aspect, a computer-implemented method for spatialized audio in a virtual metaverse is disclosed, comprising: receiving a request from a first user among a plurality of users to receive audio associated with a metaverse location of the virtual metaverse, wherein the first user is associated with a user device, and wherein the plurality of users are associated with a corresponding avatar among a plurality of avatars in the metaverse location; acquiring a data model associated with the metaverse location, wherein the data model includes one or more spatial parameters representing one or more physical laws applicable to the metaverse location; and extracting avatar information and scene information from the data model, wherein the avatar information includes one or more of the following: The location, velocity, or orientation of multiple avatars in a metaverse setting, including a first avatar associated with a first user, and wherein the scene information includes one or more of the following: occlusion, echo, or virtual wall virtually approaching the first avatar in the metaverse setting; transforming corresponding audio streams received from each of the multiple users based on the avatar information and the scene information, and transforming one or more audio features of at least one corresponding audio stream based on the aforementioned one or more spatial parameters to create a spatialized audio stream; combining the aforementioned spatialized audio streams to create a combined spatialized audio stream; and providing the combined spatialized audio stream to a user device.

[0008] This article describes various implementations of the computer-based method.

[0009] In some implementations, the spatial parameters include distance attenuation parameters for attenuating audio based on the distance between avatars.

[0010] In some implementations, the corresponding audio stream received from each of the multiple users includes mono audio received by the microphone device, and wherein the combined spatialized audio stream includes stereo audio.

[0011] In some implementations, the combined spatialized audio stream includes stereo audio generated by positioning the mono audio of each user at the location of the corresponding user's avatar.

[0012] In some implementations, the combined spatialized audio stream includes spatial audio and background audio, the spatial audio being based on audio streams received from users among a plurality of users other than the first user, wherein the background audio is generated based on one or more of the following: audio received from users other than the first user; and audio generated based on the movement of avatars within the metaverse space.

[0013] In some implementations, the computer-implemented method further includes: determining a set of prioritized audio streams received from each of a plurality of users, wherein transforming the corresponding audio streams further includes transforming the set of prioritized audio streams to create a spatialized audio stream.

[0014] In some implementations, determining a set of prioritized audio streams includes prioritizing audio streams received from each of a plurality of users based on one or more of the following: proximity of the avatar in the metaverse, speed of the avatar in the metaverse, orientation of the avatar in the metaverse, virtual objects in the metaverse that are close to the avatar, capabilities of the user device, or user preferences of the first user.

[0015] In some implementations, audio streams associated with avatars closer to the receiving avatar are preferred over audio streams associated with avatars farther from the receiving avatar, wherein audio streams associated with avatars facing the receiving avatar are preferred over audio streams associated with avatars facing away from the receiving avatar, and wherein audio streams associated with avatars moving toward the receiving avatar are preferred over audio streams associated with avatars moving away from the receiving avatar.

[0016] According to another aspect, a computer-implemented method for providing spatialized audio in a virtual metaverse is disclosed, comprising: receiving a request from a first user among a plurality of users to receive audio associated with a metaverse location in the virtual metaverse, wherein the first user is associated with a user device, and wherein the plurality of users are associated with corresponding avatars among a plurality of avatars in the metaverse location; determining a set of prioritized audio streams received from each of the plurality of users; transforming the set of prioritized audio streams to create a spatialized audio stream; combining the spatialized audio streams to create a combined spatialized audio stream; and providing the combined spatialized audio stream to the user device.

[0017] This article describes various implementations of the computer-based method.

[0018] In some implementations, determining a set of prioritized audio streams includes prioritizing audio streams received from each of a plurality of users based on one or more of the following: proximity of the avatar in the metaverse, speed of the avatar in the metaverse, orientation of the avatar in the metaverse, virtual objects in the metaverse that are close to the avatar, capabilities of the user device, or user preferences of the first user.

[0019] In some implementations, audio streams associated with avatars closer to the receiving avatar are preferred over audio streams associated with avatars farther from the receiving avatar, wherein audio streams associated with avatars facing the receiving avatar are preferred over audio streams associated with avatars facing away from the receiving avatar, and wherein audio streams associated with avatars moving toward the receiving avatar are preferred over audio streams associated with avatars moving away from the receiving avatar.

[0020] According to another aspect, a system is disclosed, comprising: a memory storing instructions; and a processing device coupled to the memory, the processing device being configured to access the memory, wherein, when executed by the processing device, the instructions cause the processing device to perform operations including: receiving a request from a first user among a plurality of users to receive audio associated with a metaverse location in a virtual metaverse, wherein the first user is associated with a user device, and wherein the plurality of users are associated with corresponding avatars among a plurality of avatars in the metaverse location; acquiring a data model associated with the metaverse location, wherein the data model includes one or more spatial parameters representing one or more physical laws applicable to the metaverse location; and extracting avatar information from the data model. The system comprises: information and scene information, wherein the avatar information includes one or more of the following: the position, velocity, or orientation of multiple avatars in the metaverse space, including a first avatar associated with a first user; and wherein the scene information includes one or more of the following: occlusion, echo, or virtual wall virtually approaching the first avatar in the metaverse space; transforming corresponding audio streams received from each of the multiple users based on the avatar information and scene information, and transforming one or more audio features of at least one corresponding audio stream based on one or more spatial parameters to create a spatialized audio stream; combining the spatialized audio streams to create a combined spatialized audio stream; and providing the combined spatialized audio stream to a user device.

[0021] This article describes various implementations of the system.

[0022] In some implementations, the spatial parameters include distance attenuation parameters for attenuating audio based on the distance between avatars.

[0023] In some implementations, the corresponding audio stream received from each of the multiple users includes mono audio received by the microphone device, and wherein the combined spatialized audio stream includes stereo audio.

[0024] In some implementations, the combined spatialized audio stream includes stereo audio generated by positioning the mono audio of each user at the location of the corresponding user's avatar.

[0025] In some implementations, the combined spatialized audio stream includes spatial audio and background audio, the spatial audio being based on audio streams received from users among a plurality of users other than the first user, wherein the background audio is generated based on one or more of the following: audio received from users other than the first user; and audio generated based on the movement of avatars within the metaverse space.

[0026] In some implementations, the operation further includes: determining a set of prioritized audio streams received from each of a plurality of users, wherein transforming the corresponding audio streams further includes transforming the set of prioritized audio streams to create a spatialized audio stream.

[0027] In some implementations, determining a set of prioritized audio streams includes prioritizing audio streams received from each of a plurality of users based on one or more of the following: proximity of the avatar in the metaverse, speed of the avatar in the metaverse, orientation of the avatar in the metaverse, virtual objects in the metaverse that are close to the avatar, capabilities of the user device, or user preferences of the first user.

[0028] In some implementations, audio streams associated with avatars closer to the receiving avatar are preferred over audio streams associated with avatars farther from the receiving avatar, wherein audio streams associated with avatars facing the receiving avatar are preferred over audio streams associated with avatars facing away from the receiving avatar, and wherein audio streams associated with avatars moving toward the receiving avatar are preferred over audio streams associated with avatars moving away from the receiving avatar.

[0029] In some implementations, the system further includes: a spatialized audio manager for transforming the corresponding audio stream received from each of the multiple users; and an audio device overlay module for disabling non-spatialized audio at the user device before providing the combined spatialized audio stream to the user device.

[0030] According to another aspect, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium stores instructions that, in response to execution by a processed device, cause the processing device to perform operations including: acquiring a data model associated with a metaverse location of a virtual metaverse, wherein the data model includes one or more spatial parameters representing a set of physical laws applicable to the metaverse location; receiving a request to join the metaverse location from a first user among a plurality of users, wherein the first user is associated with a first avatar and a user device, and wherein the plurality of users are associated with a plurality of avatars in the metaverse location; and, in response to the request, extracting avatar information and scene information from the data model. The avatar information includes one or more of the following: the position, velocity, or direction of the first avatar and multiple avatars in the metaverse space; and the scene information includes one or more of the following: occlusions, echoes, or virtual walls virtually approaching the first avatar; based on the avatar information and scene information, transforming the corresponding audio stream received from each of the multiple users using the aforementioned spatial parameters, wherein the transformation includes modifying one or more spatial parameters to create a spatialized audio stream; combining the aforementioned spatialized audio streams to create a combined spatialized audio stream; and providing the combined spatialized audio stream to the user device.

[0031] This document describes various implementations of this non-transitory computer-readable medium.

[0032] According to another aspect, systems, methods, and portions, features, and implementation details of non-transitory computer-readable media may be combined to form other aspects, including omitting and / or modifying certain or partial individual components or features, including other components or features, and / or other modifications; all such modifications are within the scope of this disclosure. Attached Figure Description

[0033] Figure 1 This is a diagram of an example network environment for providing spatialized audio chat in a virtual metaverse, according to some implementation methods.

[0034] Figure 2A This is a diagram of an example network environment for providing spatialized audio chat in a virtual metaverse, according to some implementation methods.

[0035] Figure 2B This is a diagram of an example network environment for providing spatialized audio chat in a virtual metaverse, according to some implementation methods.

[0036] Figure 3 This is a diagram of an example network environment for prioritizing spatialized audio streams in a virtual metaverse, according to some implementation methods.

[0037] Figure 4 This is a diagram illustrating an example three-dimensional metaverse location in a virtual metaverse according to some implementations.

[0038] Figure 5 This is a diagram illustrating an example three-dimensional metaverse location in a virtual metaverse according to some implementations.

[0039] Figure 6 This is a flowchart of an example method for providing spatialized audio chat in a virtual metaverse, according to some implementation methods.

[0040] Figure 7 This is a flowchart of an example method for prioritizing spatialized audio streams in a virtual metaverse, according to some implementations.

[0041] Figure 8 This is a block diagram illustrating an example computing device that can be used to implement one or more features described herein, according to some embodiments. Detailed Implementation

[0042] One or more embodiments described herein relate to spatialized audio associated with online gaming platforms. Features may include automatically sorting and providing spatialized audio based on the position, velocity, and / or other factors associated with virtual objects, avatars, and other items in a metaverse setting within a virtual metaverse.

[0043] The features described herein provide spatialized audio for output on client devices connected to an online platform (e.g., an online experience platform or an online gaming platform). The online platform can provide a virtual metaverse with multiple metaverse locations associated with it. Virtual avatars associated with a user can traverse these metaverse locations and interact with them, as well as items, characters, other avatars, and other objects within them. These avatars can move from one metaverse location to another while experiencing spatialized audio that provides a more immersive and enjoyable experience. Spatialized audio streams from multiple users (e.g., avatars associated with multiple users) can be prioritized based on numerous factors, thereby providing rich audio while taking into account the location, speed, movement, and actions of avatars and characters, as well as the bandwidth, processing power, and other capabilities of the client device.

[0044] By prioritizing and combining different audio streams, spatialized audio streams can be provided for output on client devices. This delivers a rich user experience, reduces the computational load for spatialized audio, and lowers bandwidth requirements without compromising the virtual immersive experience. Furthermore, defining a spatial audio application programming interface (API) allows users and developers to implement spatialized audio for virtually any online experience, enabling high-quality online virtual experiences, games, metaverse spaces, and other interactive experiences with immersive audio while reducing the technical skill requirements for users and developers.

[0045] Online experience platforms and online gaming platforms (also known as "user-generated content platforms" or "user-generated content systems") provide users with various ways to interact with each other. For example, users of online experience platforms can create games or other content or resources within the online platform (e.g., characters, graphics, items, etc., for playing and / or using in a virtual metaverse).

[0046] Users of online experience platforms can work towards common goals in metaverse settings, games, or game creation; share various virtual items (e.g., inventory items, game items, etc.); participate in audio chat (e.g., spatialized audio chat); and send each other electronic messages. Users of online experience platforms can interact with other users and play games (e.g., including avatars or other game objects and mechanics). Online experience platforms may also allow users to communicate with each other. For example, users of online experience platforms can communicate with each other using voice messages (e.g., voice chat via spatialized audio), text messages, video messages (e.g., including spatialized audio), or combinations thereof. Some online experience platforms can provide virtual 3D environments or multiple environments linked within the metaverse, in which users can interact with other users or play online games.

[0047] To enhance the entertainment value of an online experience platform, the platform can provide rich audio for playback on user devices. This audio can include, for example, different audio streams from different users and background audio. According to the various implementations described herein, different audio streams can be transformed into spatialized audio streams. These spatialized audio streams can be combined, for example, to provide a combined spatialized audio stream for playback on client devices. Furthermore, prioritized audio streams can be provided, thereby reducing bandwidth while still providing immersive, spatialized audio. Additionally, background audio streams can be combined with spatialized audio so that realistic background noise / effects are also played to the user. Moreover, when creating spatialized audio and / or background audio to enhance the realism and immersion in online virtual experiences, features of the metaverse are utilized, such as surrounding media (air, water, others), reverberation, reflections, aperture size, wall density, ceiling height, doorways, corridors, object placement, non-player objects / characters, and other features.

[0048] Figures 1 to 3 System Architecture

[0049] Figure 1An example network environment 100 according to some embodiments of the present disclosure is illustrated. This network environment 100 (also referred to herein as the “system”) includes an online experience platform 102, a first client device 110, and a second client device 116 (generally referred to herein as “client devices 110 / 116”) connected via a network 122. The online experience platform 102 may include a game engine 104, one or more games 105, a spatialized audio API 106, and a data storage area 108. Client device 110 may include a game application 112, and client device 116 may include a game application 118. Users 114 and 120 may interact with the online experience platform 102 and other users using the online experience platform 102, respectively, using client devices 110 and 116.

[0050] A network environment 100 is provided for illustration. In some embodiments, the network environment 100 may include interfaces with... Figure 1 The same, fewer, more, or different elements configured in the same or different ways as shown.

[0051] In some implementations, network 122 may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wired network (e.g., Ethernet), or a wireless network (e.g., an 802.11 network). Networks, or wireless LANs (WLANs), cellular networks (e.g., Long Term Evolution (LTE) networks), routers, hubs, switches, server computers, or combinations thereof.

[0052] In some implementations, data storage area 108 may be a non-transitory computer-readable storage device (e.g., random access memory), a cache, a drive (e.g., a hard disk drive), a flash drive, a database system, or another type of component or device capable of storing data. Data storage area 108 may also include multiple storage components (e.g., multiple drives or multiple databases) that may span multiple computing devices (e.g., multiple server computers).

[0053] In some implementations, the online experience platform 102 may include a server with one or more computing devices (e.g., a cloud computing system, rack server, server computer, physical server cluster, virtual server, etc.). In some implementations, the server may be included in the online experience platform 102 and may be a standalone system or part of another system or platform.

[0054] In some implementations, the online experience platform 102 may include one or more computing devices (e.g., rack servers, router computers, server computers, personal computers, mainframe computers, laptop computers, tablet computers, desktop computers, etc.), data storage areas (e.g., hard disks, memory, databases), networks, software components, and / or hardware components that can be used to perform operations on the online experience platform 102 and provide users with access to the online experience platform 102. The online experience platform 102 may also include websites (e.g., one or more web pages) or application backend software that can be used to provide users with access to content provided by the online experience platform 102. For example, users 114 / 120 may access the online experience platform 102 using game applications 112 / 118 on client devices 110 / 116, respectively.

[0055] In some implementations, the online experience platform 102 may include a social network that provides connectivity between users or a user-generated content system that allows users (e.g., end users or consumers) to communicate with other users through the online experience platform 102. In this user-generated content system, communication may include voice chat (e.g., synchronous and / or asynchronous voice communication with / without spatialized audio), video chat (e.g., synchronous and / or asynchronous video communication with / without spatialized audio), or text chat (e.g., synchronous and / or asynchronous text-based communication).

[0056] In some embodiments of this disclosure, "user" may refer to a single individual. However, other embodiments of this disclosure encompass "user" as an entity controlled by a group of users or an automated source (e.g., a creative user). For example, a group of individual users united as a community or group in a user-generated content system can be considered "users".

[0057] In some implementations, the online experience platform 102 may be a virtual gaming platform. For example, the gaming platform may offer single-player or multiplayer games to a user community that can access or interact with the game (e.g., a user-generated game or other game) via network 122 using client devices 110 / 116. In some implementations, the game (also referred to herein as a “video game,” “online game,” “metaverse space,” or “virtual experience”) may be a two-dimensional (2D) game, a three-dimensional (3D) game (e.g., a user-generated 3D game), a virtual reality (VR) game, or an augmented reality (AR) game. In some implementations, users can search for games and game items and participate in playing one or more games with other users. In some implementations, games can be played in real-time with other users in the game. Similarly, some users can engage in real-time voice or video chat with other users in the game. As described herein, real-time voice or video chat may include spatialized audio.

[0058] In some implementations, alternatives to the online experience platform 102 and / or the spatial audio API 106, or other collaborative platforms besides the online experience platform 102 and / or the spatial audio API 106, may also be used with the features described herein. For example, social networking platforms, purchasing platforms, messaging platforms, creation platforms, etc., may be used with the spatial audio features described herein to provide users with immersive spatial audio outside of gaming.

[0059] In some implementations, gameplay may refer to interactions between one or more players within a game (e.g., 105) using client devices (e.g., 110 and / or 116), or to interactions presented on a display or other output device of client device 110 or client device 116. In some implementations, gameplay may also refer to interactions within a virtual experience or metaverse setting, and may include objectives that are dissimilar, different, or the same as some games. Furthermore, although referred to as "player," the terms "avatar," "user," and / or other terms may also be used to refer to a user participating in and / or interacting with an online virtual experience.

[0060] An online experience platform provides one or more games 105. In some embodiments, game 105 may include an electronic file that can be executed or loaded using software, firmware, or hardware for presenting virtual content (e.g., digital media projects) to entities. In some embodiments, game applications 112 / 118 may be executed and combined with game engine 104 to render game 105. In some embodiments, game 105 may have a set of common rules or common goals, and the environment of game 105 shares this set of common rules or common goals. In some embodiments, different virtual experiences may have different rules or goals from each other. It should be noted that although specifically referred to as "game" or related to games, game applications 112 / 118, game 105, and game engine 104 may also be referred to as virtual experience applications 112 / 118, virtual experience 105, and / or virtual experience engine.

[0061] In some implementations, the game and / or virtual experience may have one or more environments in which multiple environments can be linked (also referred to herein as “game environment,” “metaverse location,” or “virtual environment”). An example of an environment may be a three-dimensional (3D) environment. One or more environments of the game 105 or virtual experience may be collectively referred to herein as a “world,” “game world,” “virtual world,” “universe,” or “metaverse.” An example of a world may be a 3D metaverse location of the game 105. For example, a user can construct a metaverse location linked to another metaverse location created by a different user than the first user. Characters in the virtual experience can cross virtual boundaries to enter adjacent metaverse locations. Furthermore, sounds, theme music, and / or background music can also cross virtual boundaries, allowing avatars standing near virtual boundaries to hear spatialized audio that includes at least a portion of sounds emanating from adjacent metaverse locations. In this way, spatialized audio enables a fully immersive experience that includes virtual audio representing sound propagation similar to that of a real-world environment.

[0062] It should be noted that 3D environments or 3D worlds use graphics that represent a three-dimensional representation of the content using geometric data (or at least render the content as 3D content, regardless of whether a 3D representation of geometric data is used). 2D environments or 2D worlds use graphics that represent a two-dimensional representation of the game content using geometric data.

[0063] In some implementations, the online experience platform 102 may host one or more games 105 and may allow users to interact with the games 105 using game applications 112 / 118 on client devices 110 / 116 (e.g., searching for games, game-related content, or other content). Users of the online experience platform 102 (e.g., 114 and / or 120) may play, create, interact with, or build games 105, search for games 105, communicate with other users, create and build objects of game 105 (e.g., also referred to herein as “projects” or “game objects” or “virtual game projects”), and / or search for objects. For example, in generating user-generated virtual projects, users may create characters, character decorations, one or more virtual environments for interactive gameplay, or build structures used in game 105, etc.

[0064] In some implementations, users can buy, sell, or trade virtual game objects, such as in-platform currency (e.g., virtual currency), with other users on the online experience platform 102. In some implementations, the online experience platform 102 can send game content to a game application (e.g., 112). In some implementations, game content (also referred to herein as "content") can refer to any data or software instructions (e.g., game objects, games, user information, videos, images, commands, media items, etc.) associated with the online experience platform 102 or the game application.

[0065] In some implementations, a game object (e.g., also referred to herein as a "project" or "object" or "virtual game project") may refer to an object used, created, shared, or otherwise depicted in the game application 105 of the online experience platform 102 or the game application 112 or 118 of the client devices 110 / 116. For example, a game object may include parts, models, characters, tools, weapons, clothing, buildings, vehicles, currency, flora, fauna, components of the aforementioned objects (e.g., windows of a building), etc.

[0066] It should be noted that the online experience platform 102 providing the hosting of game 105 is for illustrative purposes and not for limitation. In some embodiments, the online experience platform 102 may host one or more media items, which may include communication messages from one user to one or more other users. Media items may include, but are not limited to, digital videos, digital movies, digital photos, digital music, audio content, melodies, website content, social media updates, e-books, e-magazines, digital newspapers, digital audiobooks, e-journals, web blogs, real simple syndication (RSS) subscriptions, e-comic books, software applications, etc. In some embodiments, media items may be electronic files, which may be executed or loaded using software, firmware, or hardware for presenting digital media items to entities.

[0067] In some implementations, game 105 may be associated with a specific user or group of users (e.g., a private game) or be widely available to users of the online experience platform 102 (e.g., a public game). In some implementations, where the online experience platform 102 associates one or more games 105 with a specific user or group of users, the online experience platform 102 may use user account information (e.g., a user account identifier such as a username and password) to associate a specific user with game 105. Similarly, in some implementations, the online experience platform 102 may use developer account information (e.g., a developer account identifier such as a username and password) to associate a specific developer or group of developers with game 105.

[0068] In some implementations, the online experience platform 102 or client devices 110 / 116 may include a game engine 104 or a game application 112 / 118. Game engine 104 may include a game application similar to game application 112 / 118. In some implementations, game engine 104 may be used for the development or execution of game 105. For example, game engine 104 may include a rendering engine (“renderer”) for 2D, 3D, VR, or AR graphics, a physics engine, a collision detection engine (and collision response), a sound engine, a spatialized audio manager / engine, an audio mixer, an audio subscription exchange, audio subscription logic, an audio subscription priority orderer, a real-time communication engine, scripting capabilities, an animation engine, an artificial intelligence engine, networking capabilities, streaming capabilities, storage management capabilities, threading capabilities, scene graph capabilities, or animation video support, and other capabilities. Components of game engine 104 may generate commands that aid in calculating and rendering the game (e.g., rendering commands, collision commands, physics commands, etc.) and may transform audio (e.g., transforming mono or stereo audio into spatialized audio, etc.). In some implementations, the game applications 112 / 118 of client devices 110 / 116 can work independently and / or collaborate with the game engine 104 of the online experience platform 102.

[0069] In some implementations, both the online experience platform 102 and the client devices 110 / 116 execute game engines (104, 112, and 118, respectively). The online experience platform 102, using game engine 104, may execute some or all of the game engine functions (e.g., generating physics commands, rendering commands, spatializing audio commands, etc.), or may offload some or all of the game engine functions to the game engine 104 on the client device 110. In some implementations, there may be a different ratio between the game engine functions executed on the online experience platform 102 and the game engine functions executed on the client devices 110 and 116 for each game 105.

[0070] For example, the game engine 104 of the online experience platform 102 can be used to generate physics commands in the event of a collision between at least two game objects, while additional game engine functions (e.g., generating rendering commands or combining spatialized audio streams) can be offloaded to the client device 110. In some implementations, the ratio of game engine functions executed on the online experience platform 102 and the client device 110 can be changed (e.g., dynamically) based on gameplay conditions. For example, if the number of users participating in the gameplay of game 105 exceeds a threshold, the online experience platform 102 can execute one or more game engine functions previously executed by the client device 110 or client device 116.

[0071] For example, a user can play game 105 on client devices 110 and 116 and can send control commands (e.g., user input such as right, left, up, down, user selection, or character position, and speed information) to the online experience platform 102. After receiving control commands from client devices 110 and 116, the online experience platform 102 can send gameplay commands (e.g., position and speed information of characters participating in team gameplay, or commands such as rendering commands, collision commands, spatial audio commands, etc.) to client devices 110 and 116 based on the control commands. For example, the online experience platform 102 can (e.g., using game engine 104) perform one or more logical operations on the control commands to generate gameplay commands for client devices 110 and 116. In other instances, the online experience platform 102 can pass one or more control commands from one client device 110 to other client devices (e.g., 116) participating in game 105. Client devices 110 and 116 can use the aforementioned gameplay instructions to render the experience for display on their respective screens. Client devices 110 and 116 can also use gameplay instructions to create, modify, and / or combine spatialized audio streams for output on their respective audio output devices.

[0072] In some implementations, control commands may refer to instructions instructing a user character to perform in-game actions. For example, control commands may include user input controlling in-game actions, such as right, left, up, down, user selection, gyroscope position and orientation data, force sensor data, etc. Control commands may include character position and speed information. In some implementations, control commands are sent directly to the online experience platform 102. In other implementations, control commands may be sent from client device 110 to another client device (e.g., 116), where the other client device uses a local game engine 104 to generate gameplay instructions. Control commands may include instructions to play voice communication messages or other sounds from another user on an audio device (e.g., a speaker, headphones, etc.).

[0073] In some implementations, gameplay instructions may refer to instructions that allow client device 110 (or 116) to render gameplay of a game (e.g., a multiplayer game). Game instructions may include one or more of user input (e.g., control instructions), character position and velocity information, or commands (e.g., physics commands, rendering commands, collision commands). As described in more detail herein, character position and velocity information may be used to determine an appropriate head-related transfer function (HRTF) associated with another character, thereby creating a spatialized audio stream representing sound propagation in the real world for that character. This associated HRTF, position information, velocity information, Baum-Welch (BW) algorithm data, virtual auditory display (VAD) data, and / or other data may be stored in data storage area 108 by online experience platform 102.

[0074] In some implementations, a character (or game object in general) is composed of components, one or more of which a user can select, which are automatically linked together to aid in editing. One or more characters (also referred to herein as "avatars" or "models") can be associated with a user, who can control the character to facilitate interaction with the game 105. In some implementations, a character may include components such as body parts (e.g., head, hair, arms, legs, etc.) and accessories (e.g., T-shirts, glasses, decorative images, tools, etc.). In some implementations, customizable body parts of a character include head type, body part types (arms, legs, torso, and hands), face type, hair type, and skin type. In some implementations, customizable accessories include clothing (e.g., shirts, trousers, hats, shoes, glasses, etc.), weapons, or other tools.

[0075] In some implementations, the user can also control the size of the character (e.g., height, width, or depth) or the size of the character's components. In some implementations, the user can control the character's proportions (e.g., blocky, anatomical, etc.). It should be noted that in some implementations, the character may not include character game objects (e.g., body parts, etc.), but the user can still control the character (even without character objects) to facilitate user interaction with the game (e.g., in a puzzle game without rendered character objects, the user still controls the character to control in-game actions).

[0076] In some implementations, components such as body parts may be primitive geometries (e.g., blocks, cylinders, spheres, etc.) or other primitive shapes (e.g., wedges, rings, tubes, channels, etc.). In some implementations, the creator module may publish a user's character for other users of the online experience platform 102 to view or use. In some implementations, users may use a user interface (e.g., a developer interface) to create, modify, or customize characters, other game objects, game 105, or game environments with or without scripts (or with or without an application programming interface (API)). It should be noted that, for illustrative and not limiting purposes, characters are described as having a humanoid form. It should also be noted that characters can have any form, such as vehicles, animals, inanimate objects, or other creative forms.

[0077] In some implementations, the online experience platform 102 may store user-created characters in data storage area 108. In some implementations, the online experience platform 102 maintains a character directory and a game directory that can be presented to users via game engine 104, game 105, and / or client devices 110 / 116. In some implementations, the game directory includes images of games stored on the online experience platform 102. Furthermore, users can select characters (e.g., characters created by themselves or other users) from the character directory to participate in selected games. The character directory includes images of characters stored on the online experience platform 102. In some implementations, one or more characters in the character directory may have already been created or customized by the user. In some implementations, the selected character may have character settings that define one or more components of that character.

[0078] In some implementations, a user's role may include component configurations, wherein the component configuration and appearance (more generally, the appearance of the role) can be defined by role settings. In some implementations, the role settings of a user's role can be at least partially selected by the user. In other implementations, a user can select a role with default role settings or role settings selected by other users. For example, a user can select a default role from a directory of roles with predefined role settings, and the user can further customize the default role by changing some role settings (e.g., adding a shirt with a custom logo). Role settings can be associated with specific roles through the online experience platform 102.

[0079] In some embodiments, client device 110 or client device 116 may each include a computing device (e.g., a personal computer (PC)), a mobile device (e.g., a laptop computer, mobile phone, smartphone, tablet computer, or netbook computer), an internet television, a game console, etc. In some embodiments, client device 110 or client device 116 may also be referred to as a "user device." In some embodiments, one or more client devices 110 or client device 116 can be connected to the online experience platform 102 at any given time. It should be noted that the number of client devices 110 or client devices 116 provided is for illustrative purposes and not for limitation. In some embodiments, any number of client devices 110 or client devices 116 may be used.

[0080] In some implementations, each client device 110 or client device 116 may include an instance of game application 112 or 118. In some implementations, game application 112 or 118 may allow a user to use and interact with online experience platform 102, for example, to search for games, experiences, or other content; control virtual characters in virtual experiences hosted by online experience platform 102; or view or upload content, such as game 105, images, video items, web pages, documents, etc. In one example, the game application may be a web application (e.g., an application that operates in conjunction with a web browser) that can access, retrieve, render, or navigate content served by a web server (e.g., virtual characters in a virtual environment). In another example, the game application may be a native application (e.g., a mobile application, app, or game program) that is locally installed and executed on client device 110 or client device 116 and allows the user to interact with online experience platform 102. The game application may render, display, or present content (e.g., web pages, user interfaces, media viewers, audio streams) to the user. In implementations, the game application may also include an embedded media player embedded in a web page.

[0081] According to various aspects of this disclosure, game applications 112 / 118 can be online experience platform applications for users to build, create, edit, upload content to online experience platform 102, and interact with online experience platform 102 (e.g., play game 105 hosted by online experience platform 102). Therefore, game applications 112 / 118 can be provided by online experience platform 102 to client device 110 or client device 116. In another example, game applications 112 / 118 can be applications downloaded from a server.

[0082] In some implementations, users can log in to the online experience platform 102 via a game application. Users can access a user account by providing user account information (e.g., username and password), wherein the user account is associated with one or more characters that can be used to participate in one or more games 105 on the online experience platform 102.

[0083] Typically, in other embodiments, functions described as being performed by the online experience platform 102 may also be performed by client device 110 or client device 116 or a server, if appropriate. Furthermore, functions belonging to a particular component may be performed by different components or multiple components operating together. The online experience platform 102 can also be accessed as a service provided to other systems or devices through a suitable application programming interface (API), and is therefore not limited to use on a website.

[0084] In some implementations, the online experience platform 102 may include a spatialized audio API 106. In some implementations, the spatialized audio API 106 may be a set of computer-executable code that provides functionality to users and / or developers in the form of function calls that allow software components to transmit and / or provide / receive data. The spatialized audio API includes several defined software functions related to spatialized audio, which can be used by developers to enable spatialized audio functionality in user-created content, and may include any functionality related to audio playback on the user's device.

[0085] In at least one implementation, the Spatial Audio API 106 includes multiple functions, events, and attributes that enable spatial audio. For example, the Spatial Audio API 106 may include functions for creating and destroying voice channels. These functions can enable the creation of new voice channels associated with a specific server and / or the creation of global voice channels shared between servers within the same metaverse. These functions may also allow the deletion / destruction of previously created voice channels.

[0086] Spatialized Audio API 106 may also include functionality for adding and removing players, as well as retrieving players associated with a voice channel. These functions enable the addition of one or more given players to a specific voice channel, the removal of players from a specific voice channel, and / or the retrieval of a list of players associated with a voice channel in a metaverse setting. In some implementations, these functions can trigger events that are activated when a player joins and / or leaves a voice channel.

[0087] Spatial Audio API 106 may also include functionality for creating voice channels not associated with a specific player. In this way, these functions enable non-player characters, objects, and other virtual items to emit sounds for spatial audio streams. For example, a speaker object can be created that emits sound like a jukebox representing a specific location within a metaverse space. An avatar near the speaker object can then receive a spatial audio stream comprising a transformed audio stream including sounds from the speaker object. Sounds created by non-player characters, objects, and other virtual items can also be incorporated into a background audio stream. This background audio stream may also include sounds created by several (e.g., one or more) other avatars or player characters.

[0088] Spatial Audio API 106 may also include attributes that include parameters or properties associated with sound propagation. In this way, the aforementioned attributes may include properties such as: propagation medium (e.g., water, air, etc.), audio source (e.g., player or non-player audio source), audio volume (e.g., the volume representing the audio source), attenuation distance (e.g., the distance at which the sound begins to attenuate), maximum distance at which the sound can be heard (e.g., if the avatar is beyond this distance, the audio stream will not be included in the spatial composition), (e.g., for a particular roll-off mode) linear or logarithmic sound roll-off, playback loudness, (e.g., the connection state of the voice channel), (e.g., if the player or sound source is muted) mute state, and other attributes.

[0089] Spatial Audio API 106 may also include additional functions, variables, properties, and / or parameters that enable the use of rich, immersive spatial audio in user-created content and / or games. See below for references. Figure 2A and Figure 2B A more comprehensive description of the online experience platform 102 regarding the operation of providing spatial audio (or combined spatial audio streams) using the Spatial Audio API 106.

[0090] Figure 2A This is a diagram of an example network environment 200 (e.g., a subset of network environment 100) for providing spatialized audio chat in a virtual metaverse, according to some embodiments. Network environment 200 is provided for illustration. In some embodiments, network environment 200 may include elements for use with… Figure 2A The same, fewer, more, or different elements configured in the same or different ways as shown.

[0091] like Figure 2AAs shown, the online experience platform 102 can (e.g., via network 122, not shown) communicate with the client device 110 to receive user audio stream 232 (e.g., signal 230 from system audio input 216) from the client device 110 and provide a combined spatialized audio stream 250 to be output on the client device 110 (e.g., via system audio output 214).

[0092] Apart from Figure 1 In addition to the components shown, the online experience platform 102 may also include a media server 202 and a data model 206. Figure 1 In addition to the components shown, the client device 110 may also include an audio mixer 204, a spatial audio manager 205, and a sound engine 260.

[0093] Generally, media server 202 is a dedicated logical server used to connect and transmit audio streams (or other data) between components of network environment 100. Media server 202 can facilitate real-time communication (e.g., between client devices and online game server 102, and vice versa).

[0094] Audio mixer 204 may be a software module for extracting audio streams from multiple player or non-player objects and transforming them into spatialized audio streams. Audio mixer 204 may include audio mixer overlay component 208, echo cancellation component 210, and / or audio device module overlay component 212.

[0095] The audio mixer overlay component 208 can be used to overlay the base audio provided by the online experience platform 102, thereby enabling spatialized audio. For example, the audio mixer overlay component 208 can receive a copy 251 of the spatialized audio output 250 and the user audio stream 234, and as output, provide a single audio stream 238 to the spatialized audio manager 205, and provide a typical audio stream 236. In this way, if the audio mixer overlay component 208 is not initialized, the client device 110 can be used to provide regular, non-spatialized audio 239. Similarly, if the audio mixer overlay component is initialized (e.g., spatialized audio is enabled on the online experience platform 102 or client device 110 for a particular game 105), a single audio stream 238 for spatialization transformation is provided to the spatialized audio manager 205.

[0096] Echo cancellation component 210 can be used to remove echoes and / or other unwanted sound artifacts from an audio stream. A filtered output 232 (e.g., echo-removed from audio stream 236) can be provided from echo cancellation component 210 to media server 202. In this way, echo cancellation component 210 can establish filters or other functions to help provide a high-quality audio stream.

[0097] The audio device module overlay component 212 can be used to overwrite and / or disable the system audio output of the client device 110, causing it to output spatialized audio instead of standard system audio (e.g., if spatialized audio for the online experience platform 102 or the client device 110 is not enabled). Otherwise, when spatialized audio is not enabled, the audio device module overlay component 212 can output regular audio 239.

[0098] Spatial audio manager 205 can be a software component that takes one or more user audio streams 238 as input and transforms these streams into spatial audio streams 242 using spatial audio API 106 and associated software functions. For example, spatial audio manager 205 can transform each individual user audio stream 238 into a new, individual spatial audio stream 242. Alternatively, spatial audio manager 205 can provide individual user audio streams 238 to another component on client device 110 for spatial transformation.

[0099] Additionally, each individual user audio stream 238 can also be used to support physical commands and / or action commands associated with the corresponding avatar. In this way, each individual user audio stream 238 can be used to implement facial animation synchronized with the audio, creating a more realistic and / or more immersive experience for the user. For example, when facial animation is synchronized with spatialized audio, attenuation due to distance is achieved while having visible facial movements that allow the user to identify the avatar making the sound, further enhancing the user experience. Similarly, individual audio streams 238 and / or spatialized audio streams 242 can be interpreted to extract emotions and / or intentions. In this way, facial animation can be extracted to further enhance the user experience.

[0100] Additionally, each individual user audio stream 238 can be used for user moderation on the online experience platform 102. For example, since each individual audio stream is already separate, it is easier to associate abusive or vulgar language with the relevant user. Users associated with abusive behavior can then be effectively moderated using a "mute" function or by removing function calls from the voice channel (via API 106). Moderation can be scalable and / or applicable to machine learning techniques to allow automated moderation tools to analyze vocal behavior, tone changes, screaming, and / or use natural language processing techniques to identify abusive behavior and automatically moderate associated users.

[0101] Data model 206 may include multiple spatial parameters related to audio transformation. For example, data model 206 may include one or more spatial parameters representing a set of physical laws applicable to a metaverse setting. These physical laws may represent or simulate a real-world sound propagation environment, may represent an exaggerated real-world sound propagation environment, and / or may represent a sound propagation environment defined in a new way. Developers can define sound propagation parameters via the open spatialized audio API 106 by assigning specific values ​​to the parameters. For example, different propagation media, roll-off audio parameters, distance attenuation features / parameters, and / or reflection parameters can be defined. Similarly, volume parameters, minimum / maximum audible parameters, and / or propagation parameters can be defined.

[0102] Data model 206 may also include avatar and / or scene information provided by the developer and the actual location of the avatar within the metaverse. Avatar information may include one or more of the following: the position, velocity, or orientation of the avatar within the metaverse. Scene information may include one or more of the following: occlusions, echoes, virtual objects, non-player objects, openings, apertures, reflective surfaces, virtual ceilings, virtual floors, virtual corridors, virtual doorways, and / or virtual walls that are virtually close to the avatar within the metaverse. Scene information may also include information related to the medium of the surrounding environment (e.g., water, air, etc.). This information / data may be separated based on each avatar within the metaverse (e.g., as personalized space, avatar, and scene information signal 244) and provided to the spatialized audio manager 205 and / or the corresponding client devices 110 / 116 for transformation.

[0103] Subsequently, sound engine 260 (or optionally, spatial audio manager 205 and / or game engine 104) can combine multiple spatial audio streams 246 into a combined spatial audio stream 250 for output on client device 110. Sound engine 260 may include any suitable sound engine, including a sound effects engine and / or a portion of game engine 104 dedicated to audio effects. In at least one embodiment, sound engine 260 is a proprietary audio effects engine. In other embodiments, sound engine 260 may be a digital effects engine or a game engine.

[0104] In the following text, see references Figure 2B Describe the optional network environment 275. Figure 2B This is a diagram of an example network environment 275 (e.g., a subset of network environment 100) for providing spatialized audio chat in a virtual metaverse, according to some embodiments. Network environment 275 is provided for illustration. In some embodiments, network environment 275 may include interfaces with... Figure 2B The same, fewer, more, or different elements configured in the same or different ways as shown.

[0105] like Figure 2B As shown, the online experience platform 102 can (e.g., via network 122, not shown) communicate with client device 110, enabling it to receive user audio stream 232 (e.g., signal 230 from system audio input 216) from client device 110 and provide a combined spatialized audio stream 250 to be output on client device 110 (e.g., via system audio output 214). Note that, for the sake of brevity, the components and / or portions of network environment 275 that have the same numbering as the components and / or portions of network environment 200 will not be described again here.

[0106] Similar to Figure 2A The online experience platform 102 shown may include a media server 202 and a data model 206. Similar to... Figure 2A The client device 110 shown may include an audio mixer 204 and a spatialized audio manager 205. However, with Figure 2A Conversely, the spatial audio manager 205 can provide output 250 based on data received from the game engine 104, a single audio stream 238, and individual spatial, avatar, and scene information signals 244 (e.g., combined signals 276). In this way, the spatial audio manager 205 can include sound engine functionality embedded in or implemented within the manager. Alternatively, one or more independent sound engine components can be used to provide the spatial audio output 250.

[0107] In this way, the spatial audio manager 205 provides a combined spatial audio stream 250 based on data received from the game engine 104 and the data model 206. For example, the spatial audio manager 205 may receive each individual user audio stream 238 as a new, single spatial audio stream 276 based on (e.g., combined at the audio confluence) spatial, avatar, and scene information signals 244.

[0108] As referenced above Figure 2A and Figure 2B As described, multiple spatialized audio streams 246 / 276 (or single, non-spatialized audio streams 238) can be combined to produce a combined spatialized audio stream 250 for output on a specific client device 110. Considering that any given virtual metaverse or virtual environment may have a very large number of available audio streams, some implementations provide a prioritization of audio streams. Prioritization of audio streams (sometimes referred to as a “subscription” to audio streams) allows a reduced set of prioritized streams (e.g., compared to all available streams) to be transformed into a spatialized audio stream. This reduced set of streams provides technical benefits and effects, including reduced computation cycles for generating the combined spatialized audio stream, reduced system resource usage, energy savings, and reduced bandwidth usage.

[0109] In the following text, see references Figure 3 A more comprehensive description of prioritizing spatial audio streams using the Spatial Audio API 106 and available avatar and scene data.

[0110] Figure 3 This is a diagram of an example network environment 300 (e.g., a subset of network environment 100) for prioritizing spatialized audio streams in a virtual metaverse, according to some embodiments. Network environment 300 is provided for illustration. In some embodiments, network environment 300 may include elements that can be used with… Figure 3 The same, fewer, more, or different elements configured in the same or different ways as shown.

[0111] like Figure 3 As shown, media server 202 and client device 110 can communicate (e.g., via network 122, not shown) to receive published audio stream 330 from client device 110 and to provide a set of prioritized audio streams 350 for transformation, combination, and output on client device 110. Note that in some embodiments, the prioritized audio streams 350 may be transformed and combined on online experience platform 102 instead of client device 110. All such modifications are within the scope of this disclosure.

[0112] Apart from Figure 1 and Figures 2A to 2BIn addition to the components shown, client device 110 may include real-time communication module 302 and subscription logic 312. Figure 1 and Figures 2A to 2B In addition to the components shown, the media server 202 may include a real-time communication module 306, a subscription exchange component 304, and a subscription priority sorter 310.

[0113] Real-time communication modules 302 / 306 may be software components or instances of real-time communication servers instantiated on client device 110 and media server 202, respectively. In at least one embodiment, real-time communication modules 302 / 306 are instances of WebRTC servers implemented according to an open WebRTC API (not shown). Real-time communication modules 302 / 306 are used to deliver published streams 330 from each connected client device and to deliver a set of prioritized audio streams 350 to each connected client device.

[0114] The subscription exchange component 304 is a software component for receiving at least a portion of the HRTF and / or BW Hidden Markov Model 332 / 333 estimates from the real-time communication module 306 / 302, and for prioritizing a set of audio streams to a group 331 / 351 for output to client devices.

[0115] The subscription priority orderer 310 obtains suitable and / or relevant spatial parameters 336 from the data model 206 and multiple subscription requests 334 from the spatialized audio API 106. The subscription priority orderer uses the spatial parameters and subscription requests to prioritize available audio streams into priority groups 350. Prioritization can be based on multiple factors, including, for example, proximity of avatars in the metaverse, velocity of avatars in the metaverse, orientation of avatars in the metaverse, virtual objects near avatars in the metaverse, user device capabilities, bandwidth availability, number of connections to media server 202, total number of users using spatialized audio, and / or user preferences associated with client device 110. Additional factors may include (e.g., available bandwidth between media server 202 and client device 110), (e.g., available processing power of media server 202, client device 110, and / or a combination thereof), (e.g., available memory of media server 202, client device 110, and / or a combination thereof), and other factors.

[0116] Subscription requests are based on subscription logic 312, which is used to issue individual subscription requests 338 based on spatial parameters 340 about a specific avatar of a particular user. In this way, each individual client device issues different subscription requests 338 based on the avatar and spatial parameters 340 associated with that device, as well as other factors 342 (e.g., which may include HRTF parameters, BW Hidden Markov Model parameters, and / or VAD parameters).

[0117] As described above, a set of priority flows can be based on multiple spatial parameters associated with the avatars, items, objects, and other characteristics of the metaverse site, as well as computational resources, storage resources, bandwidth resources, and other resources. In the following text, references... Figure 4 and Figure 5 Provide examples related to spatialized audio.

[0118] Figure 4 and Figure 5 Spatialized audio examples in the metaverse

[0119] Figure 4 and Figure 5 This is a diagram illustrating example virtual environments such as metaverse location 400 and metaverse location 500 in an online virtual experience according to some embodiments. Metaverse location 400 includes a first avatar 402, a second avatar 404, and a third avatar 406. Avatars (402 to 406) may include corresponding user-controlled avatars and / or automatically controlled avatars of the online experience platform 102 (e.g., computer-generated characters).

[0120] In addition to avatars 402 to 406, the metaverse space 400 also includes a first virtual object 408, a second virtual object 410, a third virtual object 412, and a fourth virtual object 422. Virtual objects can represent buildings, building components (e.g., walls, windows, doors, etc.), bodies of water (e.g., ponds, rivers, lakes, oceans, etc.), furniture, machines, vehicles, plants, animals, etc. Virtual objects (408 to 412) can include associated data or metadata corresponding to one or more object characteristics (e.g., material type (e.g., metal, wood, cloth, stone, etc.), object location in the metaverse space 400, object size, object shape, or object sound characteristics (e.g., ambient sounds emitted by the object, frequency of the sound, volume of the sound, etc.)).

[0121] Object sound characteristics can be based on object type, object size, object shape, or object location. For example, an object representing an adult dog could have typical sound characteristics of a large adult dog (object size) and be located at a given location relative to the character (object location). Sound characteristics can include ambient sounds emitted by the object (e.g., barking) that can be provided by a sound file (e.g., computer-generated or recorded sound). Furthermore, sound characteristics can include the frequency of the sounds emitted by the object (e.g., the frequency of a dog's bark) and the volume of the ambient sounds (e.g., the volume of a dog's bark at a given distance). As part of audio spatialization processing, the volume of the object's ambient sounds can then be modified (e.g., making the dog's bark louder for a dog close to the character and quieter for a dog farther away from the character).

[0122] exist Figure 4 In the example shown, objects can include furniture or parts of a building. For example, virtual object 408 could be a doorway or a virtual boundary between metaverse spaces. Virtual objects 410 and 412 could be walls. Virtual object 422 could be a small table or other virtual furniture.

[0123] As shown in sound paths 414 and 416, avatar 402 can speak and emit simulated sounds. The sound of avatar 402 comes from the left side of avatar 406 along sound path 414, and from the right side of avatar 408 along sound path 416 through virtual doorway 408. The sound path between avatars 402 and 406 is generally direct, while, as shown in sound path 416, the sound path from avatar 402 to avatar 404 is partially reflected from walls 410 and 412.

[0124] In addition, ambient sounds may be emitted by table 422. For example, the table may include a speaker or other objects on the table that emit sound. Ambient sounds are illustrated by sound paths 418 and 419. In some embodiments, ambient sounds may include sounds such as wind, rain, music, machinery, animals, moving objects, footsteps, etc. Ambient sounds may also include sounds emitted from objects (e.g., cars), which may be stationary or moving within the environment. Ambient sounds may also include sounds created by other avatars that move or interact with objects in environment 400.

[0125] In practice, the implementation of the audio spatialization technique described in this paper can be based on... Figure 4The example metaverse shown performs one or more of the following operations: 1) spatializes the voice communication of an avatar (e.g., avatar 402) based on the position of the corresponding receiving avatar (e.g., 404 or 406) relative to the speaking avatar; 2) spatializes the audio of the voice communication based on any object in the virtual environment (e.g., 408, 410, 412, or 422); 3) spatializes the audio of ambient sounds in the virtual environment (e.g., sounds emitted by object 410).

[0126] Now go to Figure 5 The figure shows a top view of the metaverse location 500. As shown, simplified schematics of avatars 502, 504, 506, and 508 include a distance radius 525 measured relative to avatar 502. In this example, avatar 502 may represent a user requesting spatialized audio, and the distance radius 525 may be a parameter or setting used for prioritizing audio streams.

[0127] As further illustrated, the arrows emanating from the corresponding avatars represent avatar position and velocity data 541, 561, and 581. In this example, the user device associated with avatar 502 can receive the prioritized audio streams associated with avatars 504 and 506, as well as ambient sounds associated with any objects or non-player items within radius 525. However, if avatar 508 continues to approach radius 525 as indicated by arrow 581, the data associated with avatar 508 can also be prepared for prioritization. Optionally, if computational resources allow or if other parameters permit, the data associated with avatar 508 can also be included in the prioritized audio stream until resources or other priority parameters change (e.g., another avatar approaches avatar 502, computational resource usage increases beyond a threshold, other audio streams such as background audio have higher priority, etc.).

[0128] In this way, a subset of available audio streams is prioritized, thereby reducing computational resources while still effectively delivering rich, immersive audio to client devices.

[0129] In the following text, see references Figure 6 and Figure 7 Provides a more detailed discussion of creating spatialized audio streams and prioritizing audio streams.

[0130] Figure 6 Example methods for creating spatialized audio streams

[0131] Figure 6 This is a flowchart of an example method 600 for creating spatialized audio in a metaverse setting, according to some implementations. In some implementations, method 600 can be implemented, for example, on a server system (e.g., such as...). Figure 1The online experience platform 102 shown is implemented. In some embodiments, some or all of method 600 can be implemented on, for example, the online experience platform 102 shown. Figure 1 The system is implemented on one or more client devices 110 and 116, and / or on a server system and one or more client systems. In the described example, the implemented system includes one or more processors or processing circuitry, and one or more storage devices such as a database or other accessible storage. In some implementations, different components of one or more servers and / or clients may perform different blocks or other portions of method 600. Method 600 may begin at block 602.

[0132] In box 602, a request to receive audio associated with a metaverse location in the virtual metaverse can be received (e.g., from a first user among multiple users). For example, client device 110 (also referred to as a user device) can be associated with the first user. The first user is associated with a first avatar. Furthermore, multiple users can be associated with multiple avatars in the metaverse location (e.g., other avatars associated with the metaverse location). Box 602 is followed by box 604.

[0133] In box 604, a data model associated with the metaverse space is obtained. For example, data model 206 may be stored in data storage 108. The data model may include one or more spatial parameters representing a set (e.g., one or more) of physical laws applicable to the metaverse space. These physical laws may be exaggerated (compared to physical laws applicable on Earth) to enhance the spatial effect, or may be attenuated to achieve only a slight spatial effect. The parameters of sound propagation and the fundamental physical laws may be adjusted and / or altered via the spatialized audio API 106 as described above. Box 604 is followed by box 606.

[0134] In box 606, in response to a request, avatar information and scene information are extracted from the data model. For example, a request may be associated with a specific client device and therefore with a specific avatar. Thus, the extracted avatar information includes one or more of the following: the position, velocity, or orientation of the specific avatar, and the positions, velocities, or orientations of multiple avatars in the metaverse space that are near the specific avatar. Similarly, scene information includes one or more of the following: occlusions, echoes, virtual objects, non-player objects, openings, apertures, reflective surfaces, virtual ceilings, virtual floors, and / or virtual walls that are virtually near the specific avatar. Box 606 is followed by box 608.

[0135] In box 608, extracted spatial parameters are used to transform the corresponding audio stream received from each of the multiple users. This transformation is based on avatar information and scene information. The transformation may include modifying one or more audio features to create a spatialized audio stream. For example, attenuation based on a distance attenuation parameter, used to attenuate audio between avatars based on distances defined in the data model, can be used to change audio features. Similarly, (e.g., using "fade in" or "roll out" effects) audio roll-in or roll-out can be implemented to change audio features. Additionally, volume increase, volume decrease, Doppler shift, reverberation, or reflection, and / or other features may be altered. In this way, the transformation outputs a spatialized audio stream for each individual avatar and / or virtual object / item within the metaverse scene. Box 608 is followed by box 610.

[0136] In block 610, spatialized audio streams are combined to create a combined spatialized audio stream. This combination can be performed on online experience platform 102, on client devices 110 / 116, and / or through a combination of online experience platform and client devices. In some embodiments, the combination can be performed on only a set of prioritized audio streams. In other embodiments, a subset of the available audio streams is transformed based on proximity or threshold distance to the avatar within the metaverse space. Other changes and limitations on the number of audio streams transformed are also possible, and such changes are within the scope of this disclosure.

[0137] According to at least one implementation, the background audio stream is also combined with the spatialized audio stream to create a combined spatialized audio stream. For example, a background or “special” stream that mixes multiple participants into background noise / chat can provide a more realistic experience. For example, in a room where 50 people are talking, an avatar may be having a conversation with several nearby avatars. While the audio from neighboring participants may be the clearest, background chat is also present around the avatar (e.g., pure silence is not realistic). Therefore, the background stream can be premixed to include, in a relatively concise manner, a common background chat from the remaining 50 avatars (e.g., a uniform background stream from all participants, which will be used by the combined stream provided to all participants). In this way, background audio can be generated based on audio received from users other than the first user, audio generated based on the movement of avatars within the metaverse space, and / or one or more of the common or “special” background streams. Box 610 is followed by box 612.

[0138] In block 612, a combined spatial audio stream is provided to a user equipment via an audio output device (e.g., a set of speakers or headphones) connected to the user equipment. In some embodiments, the spatial audio stream may be provided via an audio output device such as a virtual reality headset, augmented reality headset, head-mounted device, etc.

[0139] Boxes 602 through 612 may be performed (or repeated) in a different order than described above, and / or one or more boxes may be omitted. For example, data extraction (boxes 604 through 606) may be performed independently of audio transformation and combination (boxes 608 through 612). Furthermore, in some embodiments, receiving requests, extracting relevant data, and transforming audio may be performed in parallel or by different components.

[0140] In the following text, see references Figure 7 A more detailed discussion of flow priority ordering is provided.

[0141] Figure 7 Example methods for prioritizing spatialized audio streams

[0142] Figure 7 This is a flowchart of an example method 700 for providing content to a user based on classification, according to some implementations. In some implementations, method 700 can be implemented, for example, on a server system (e.g., such as...). Figure 1 The online experience platform 102 shown is implemented. In some implementations, some or all of method 700 can be implemented on platforms such as... Figure 1 The system is implemented on one or more client devices 110 and 116, and / or on a server system and one or more client systems. In the described example, the implemented system includes one or more processors or processing circuitry, and one or more storage devices such as a database or other accessible storage. In some implementations, different components of one or more servers and / or clients may perform different blocks or other portions of method 700. Method 700 may begin at block 702.

[0143] In box 702, a request to receive audio associated with a metaverse location in the virtual metaverse can be received (e.g., from a first user among multiple users). For example, client device 110 can be associated with the first user. The first user is associated with a first avatar. Furthermore, multiple users can be associated with multiple avatars in the metaverse location (e.g., other avatars associated with the metaverse location). Box 702 is followed by box 704.

[0144] In box 704, a set of prioritized audio streams is determined for the first user. This set of prioritized audio streams is a sorted subset of all audio streams received by each of the multiple users within the metaverse setting, as well as all other audio streams (e.g., ambient sounds, non-player character / item sounds, etc.). Prioritization may be based on, for example, threshold distance and / or threshold radius (e.g., as...). Figure 5 (As shown).

[0145] This prioritization can also be based on: the proximity of the avatar in the metaverse, the velocity of the avatar in the metaverse, the orientation of the avatar in the metaverse, virtual objects in the metaverse that are close to the avatar, the capabilities of the user's device, or the user preferences of the first user. For example, this prioritization can consider avatars moving toward the target avatar, avatars facing the target avatar, and other similar prioritization parameters.

[0146] This prioritization can also be based on the processing resources and / or capabilities of the client device. For example, prioritization can consider memory usage, disk usage, bandwidth availability, processor usage, and other resource usage to determine if there are sufficient resources to process a specific number of spatialized audio streams. The streams can then be prioritized to avoid conflicts or the use of more resources than a threshold.

[0147] This prioritization can also be based on the media server's processing resources and / or capabilities. For example, prioritization can consider memory usage, storage area usage, bandwidth availability, processor usage, the number of active connections, the number of inactive connections, the total number of users, and other resource usage to determine if there are sufficient resources to process a specific number of spatialized audio streams. The streams can then be prioritized to avoid conflicts or using more resources than a threshold.

[0148] This prioritization can also be based on the processing resources and / or capabilities of the online experience platform. For example, prioritization can consider memory usage, storage area usage, bandwidth availability, processor usage, the number of active connections, the number of inactive connections, the total number of users, the number of active online experiences, and other resource usage to determine if there are sufficient resources to process a specific number of spatialized audio streams. The streams can then be prioritized to avoid conflicts or the use of more resources than a threshold.

[0149] Other variations based on priority ordering are also possible and are within the scope of this disclosure. Box 704 is followed by Box 706.

[0150] In box 706, extracted spatial parameters are used to transform the corresponding audio stream of the prioritized audio stream. This transformation is based on avatar information and scene information. The transformation may include modifying one or more audio features to create a spatialized audio stream. For example, attenuation based on a distance attenuation parameter can be used to change audio features, which is used to attenuate audio between avatars based on distances defined in the data model. Similarly, rolling in or rolling out of audio can be implemented to change audio features. Additionally, volume increase, volume decrease, Doppler shift, reverberation, reflection, and other features can be altered. In this way, the transformation outputs a spatialized audio stream for each individual avatar and / or virtual object / item associated with the prioritized audio stream. Box 706 is followed by box 708.

[0151] In box 708, spatialized audio streams are combined to create a combined spatialized audio stream. This combination can be performed on the online experience platform 102, on client devices 110 / 116, and / or through a combination of the online experience platform and client devices. As described above, special or background audio streams can also be combined to provide ambient and / or background audio for the spatialized audio output. This combination can be performed only on that set of prioritized audio streams. Box 708 is followed by box 710.

[0152] In box 710, a combined spatialized audio stream is provided to a user equipment via an audio output device (e.g., a set of speakers or headphones) connected to the user equipment.

[0153] Boxes 702 to 710 may be executed (or repeated) in a different order than described above and / or one or more boxes may be omitted. Methods 600 and / or 700 may be executed on a server (e.g., 102) and / or a client device (e.g., 110 or 116). Furthermore, depending on any desired implementation, portions of methods 600 and / or 700 may be combined and executed sequentially or in parallel.

[0154] As described above, systems, methods, and computer-readable media can provide spatialized audio in virtual experiences. By providing a robust spatialized audio API, developers can use spatialized audio in virtually any virtual experience they create. The immersive quality of typical virtual experiences supported by the spatialized audio API provides a rich user experience, enhances user engagement (e.g., through audio positioning), provides intuitive feedback, and significantly reduces the complexity of implementing spatialized audio.

[0155] In the following text, refer to Figure 8 A more detailed description can be used for implementation. Figures 1 to 3 Various computing devices are shown.

[0156] Figure 8This is a block diagram of an example computing device 800 according to some implementations, which can be used to implement one or more features described herein. In one example, device 800 can be used to implement a computer device (e.g., Figure 1 The computing device 800 may be any suitable computer system, server, or other electronic or hardware device. For example, the computing device 800 may be a mainframe computer, desktop computer, workstation, portable computer, or electronic device (portable device, mobile device, cellular phone, smartphone, tablet computer, television, set-top box, personal digital assistant (PDA), media player, gaming device, wearable device, etc.). In some embodiments, the device 800 includes a processor 802, a memory 804, an input / output (I / O) interface 806, and audio / video input / output devices 814 (e.g., display screen, touch screen, display goggles or glasses, audio speaker, headphones, microphone, etc.).

[0157] Processor 802 can be one or more processors and / or processing circuitry to execute program code and control the basic operations of device 800. "Processor" includes any suitable hardware and / or software system, mechanism, or component that processes data, signals, or other information. A processor can include a system with a general-purpose central processing unit (CPU), multiple processing units, dedicated circuitry for implementing functions, or other systems. Processing is not limited to a specific geographical location or time. For example, a processor can perform its functions in a "real-time," "offline," or "batch processing mode." Parts of the processing can be executed by different (or the same) processing systems at different times and locations. A computer can be any processor that communicates with memory.

[0158] Memory 804 is typically disposed in device 800 for access by processor 802 and can be any suitable processor-readable storage medium, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc. Memory 804 is suitable for storing instructions for execution by the processor and is decoupled from and / or integrated with processor 802. Memory 804 may store software operated by processor 802 on server device 800, including operating system 808, application 810, and associated data 812. In some embodiments, application 810 may include instructions that enable processor 802 to perform the functions described herein, for example... Figure 6 and Figure 7 Some or all of the methods.

[0159] For example, memory 804 may include software instructions for prioritizing and / or providing spatialized audio within an online experience platform (e.g., 102) or a metaverse setting. Any software in memory 804 may optionally be stored in any other suitable storage location or on a computer-readable medium. Furthermore, memory 804 (and / or other connected storage devices) may store instructions and data used in the features described herein. Memory 804 and any other type of storage (disk, optical disk, magnetic tape, or other tangible media) may be considered as “memory” or “storage device”.

[0160] I / O interface 806 provides functionality that enables server device 800 to interface with other systems and devices. For example, network communication devices, storage devices (e.g., memory and / or data storage area 108), and input / output devices can communicate via interface 806. In some embodiments, the I / O interface can connect to interface devices including input devices (keyboard, pointing device, touch screen, microphone, camera, scanner, etc.) and / or output devices (display device, speaker device, printer, motor, etc.).

[0161] For ease of explanation, Figure 8A box is shown for each of processor 802, memory 804, I / O interface 806, software blocks 808 and 810, and database 812. These boxes may represent one or more processors or processing circuits, operating systems, memory, I / O interfaces, applications, and / or software modules. In other embodiments, device 800 may not have all of the components shown and / or may have elements that replace or are not shown herein, or other types of elements. While online experience platform 102 is described as performing the operations described in some embodiments herein, any suitable component or combination of components of online experience platform 102 or similar systems, or any suitable one or more processors associated with such a system, may perform the described operations.

[0162] User equipment can also implement and / or be used with the features described herein. Example user equipment may be a computer device including components similar to device 800, such as processor 802, memory 804, and I / O interface 806. Operating systems, software, and applications suitable for the client device may be provided in memory and used by the processor. The I / O interface for the client device may connect to network communication devices and input and output devices, such as microphones for capturing sound, cameras for capturing images or video, audio speaker devices for outputting sound, display devices for outputting images or video, or other output devices. For example, a display device within audio / video input / output device 814 may be connected to (or included in) device 800 to display pre-processed and post-processed images as described herein, wherein such a display device may include any suitable display device, such as an LCD, LED, or plasma display screen, CRT, television, monitor, touchscreen, 3D display screen, projector, or other visual display device. Some embodiments may provide an audio output device, such as a synthesized voice output or text reading.

[0163] The methods, blocks, and / or operations described herein may be performed in a different order than those shown or described, and / or concurrently (partially or completely) with other blocks or operations where appropriate. Some blocks or operations may be performed on a portion of data, and then later, for example, on another portion of data. Not all described blocks and operations need to be performed in all implementations. In some implementations, blocks and operations may be performed multiple times in a different order and / or at different times within a method.

[0164] In some implementations, some or all of the methods may be executed on a system such as one or more client devices. In some implementations, one or more methods described herein may be implemented, for example, on a server and / or on a server system and a client system. In some implementations, different components of one or more servers and / or clients may perform different blocks, operations, or other parts of the methods.

[0165] One or more methods described herein (e.g., methods 600 and / or 700) can be implemented by computer program instructions or code executable on a computer. For example, the code can be implemented by one or more digital processors (e.g., microprocessors or other processing circuitry) and can be stored on a computer program product including a non-transitory computer-readable medium (e.g., a storage medium), such as magnetic, optical, electromagnetic, or semiconductor storage media, including semiconductor or solid-state memories, magnetic tape, removable computer floppy disks, random access memory (RAM), read-only memory (ROM), flash memory, rigid disks, optical disks, solid-state storage drives, etc. Program instructions can also be contained in and provided as electronic signals, for example, in the form of software as a service (SaaS) delivered from a server (e.g., a distributed system and / or cloud computing system). Alternatively, one or more methods can be implemented in hardware (logic gates, etc.) or a combination of hardware and software. Example hardware can be a programmable processor (e.g., a field-programmable gate array (FPGA), a complex programmable logic device), a general-purpose processor, a graphics processor, an application-specific integrated circuit (ASIC), etc. One or more methods can be executed as part of or a component of an application running on the system, or as an application or software that runs with other applications and the operating system.

[0166] One or more methods described herein can run in standalone programs that can run on any type of computing device, programs that run on a web browser, or mobile applications (“apps”) that run on mobile computing devices (e.g., mobile phones, smartphones, tablets, wearable devices (watches, armbands, jewelry, headwear, goggles, glasses, etc.), laptops, etc.). In one example, a client / server architecture can be used, whereby the mobile computing device (as a client device) sends user input data to a server device and receives final output data from the server for output (e.g., for display). In another example, all computations can be performed within a mobile application (and / or other applications) on the mobile computing device. In yet another example, computations can be split between the mobile computing device and one or more server devices.

[0167] While specific embodiments have been described herein, these embodiments are for illustrative purposes only and not for limitation. The concepts illustrated in the examples can be applied to other examples and embodiments.

[0168] In some implementations discussed herein where user data (e.g., user demographics, user behavior data on the platform, user search history, purchased and / or viewed items, user friendships on the platform, etc.) can be obtained or used, users are provided with options to control whether, how, such information is collected, stored, or used. That is, the implementations discussed herein collect, store, and / or use user information only when explicit user authorization is received and applicable regulations are met.

[0169] Provide users with control over whether a program or feature collects user information about that specific user or other users in relation to that program or feature. Present options (e.g., through a user interface) to each user for whom information is to be collected, allowing the user to exercise control over the collection of information related to that user, providing permission or authorization regarding whether information is collected and which parts of the information will be collected. Furthermore, some data may be modified in one or more ways before being stored or used, such that personally identifiable information is removed. As an example, a user's identity may be modified (e.g., by using pseudonyms, numerical values, etc.) to make personally identifiable information unidentifiable. In another example, a user's geographic location may be generalized to a larger area (e.g., city, postal code, state, country, etc.).

[0170] Note that the functional blocks, operations, features, methods, devices, and systems described in this disclosure can be integrated or divided into different combinations of systems, devices, and functional blocks known to those skilled in the art. Routines for a particular implementation can be implemented using any suitable programming language and programming techniques. Different programming techniques can be employed, such as procedural or object-oriented. The routine can be executed on a single processing device or multiple processors. Although steps, operations, or calculations are presented in a specific order, this order can be varied in different specific implementations. In some implementations, multiple steps or operations shown as being performed sequentially in this specification can be performed simultaneously.

Claims

1. A computer-based method for spatialized audio in a virtual metaverse, comprising: A request is received from a first user among a plurality of users to receive audio associated with a metaverse location in the virtual metaverse, wherein the first user is associated with a user device, and wherein the plurality of users are associated with a corresponding avatar among a plurality of avatars in the metaverse location; Obtain a data model associated with the metaverse site, wherein the data model includes one or more spatial parameters representing one or more physical laws applicable to the metaverse site; Avatar information and scene information are extracted from the data model, wherein the avatar information includes one or more of the following: the position, velocity, or direction of the plurality of avatars in the metaverse location, the plurality of avatars including the first avatar associated with the first user, and wherein the scene information includes one or more of the following: occlusions, echoes, or virtual walls that are virtually close to the first avatar in the metaverse location; Based on the avatar information and the scene information, the corresponding audio stream received from each of the plurality of users is transformed, and based on the one or more spatial parameters, one or more audio features of at least one of the corresponding audio streams are transformed to create a spatialized audio stream; Determine a set of prioritized audio streams received from each of the plurality of users, wherein transforming the corresponding audio streams further includes transforming the set of prioritized audio streams to create the spatialized audio stream, wherein audio streams associated with an avatar moving toward a receiving avatar are prioritized over audio streams associated with an avatar moving away from the receiving avatar. Combine the spatialized audio streams to create a combined spatialized audio stream; and The combined spatialized audio stream is provided to the user equipment.

2. The computer-implemented method according to claim 1, wherein, The spatial parameters include distance attenuation parameters used to attenuate audio based on the distance between avatars.

3. The computer-implemented method according to claim 1, wherein, The corresponding audio stream received from each of the plurality of users includes mono audio received by a microphone device, and wherein the combined spatialized audio stream includes stereo audio.

4. The computer-implemented method according to claim 3, wherein, The combined spatialized audio stream includes stereo audio generated by positioning each user's mono audio at the location of the corresponding user's avatar.

5. The computer-implemented method according to claim 1, wherein, The combined spatialized audio stream includes spatial audio and background audio, the spatial audio being based on the audio stream received from one of the plurality of users other than the first user, wherein the background audio is generated based on one or more of the following: Audio received from users other than the first user; and Audio generated based on the movement of the avatar within the metaverse space.

6. The computer-implemented method according to claim 1, wherein, Determining the set of prioritized audio streams includes: Based on the velocity of the avatar in the metaverse, the audio streams received from each of the plurality of users are prioritized.

7. The computer-implemented method according to claim 6, wherein, Determining the set of prioritized audio streams includes: The audio streams received from each of the plurality of users are prioritized based on one or more of the following: proximity of the avatar in the metaverse, orientation of the avatar in the metaverse, virtual objects in the metaverse that are close to the avatar, the capabilities of the user device, or the user preferences of the first user.

8. The computer-implemented method according to claim 1, wherein, The audio stream associated with the avatar closer to the receiving avatar takes precedence over the audio stream associated with the avatar farther from the receiving avatar, and wherein the audio stream associated with the avatar facing the receiving avatar takes precedence over the audio stream associated with the avatar facing away from the receiving avatar.

9. A computer-implemented method for providing spatialized audio in a virtual metaverse, comprising: A request is received from a first user among a plurality of users to receive audio associated with a metaverse location in the virtual metaverse, wherein the first user is associated with a user device, and wherein the plurality of users are associated with a corresponding avatar among a plurality of avatars in the metaverse location; A set of prioritized audio streams received from each of the plurality of users is determined, wherein audio streams associated with an avatar moving toward the receiving avatar are prioritized over audio streams associated with an avatar moving away from the receiving avatar; Transform the set of prioritized audio streams to create a spatial audio stream; Combine the spatialized audio streams to create a combined spatialized audio stream; and The combined spatialized audio stream is provided to the user equipment.

10. The computer-implemented method according to claim 9, wherein, Determining the set of prioritized audio streams includes: The audio streams received from each of the plurality of users are prioritized based on one or more of the following: proximity of the avatar in the metaverse, speed of the avatar in the metaverse, direction of the avatar in the metaverse, virtual objects in the metaverse that are close to the avatar, the capabilities of the user device, or the user preference of the first user.

11. The computer-implemented method according to claim 9, wherein, The audio stream associated with the avatar closer to the receiving avatar takes precedence over the audio stream associated with the avatar farther from the receiving avatar, and wherein the audio stream associated with the avatar facing the receiving avatar takes precedence over the audio stream associated with the avatar facing away from the receiving avatar.

12. A system for spatialized audio in a virtual metaverse, comprising: A memory, on which instructions are stored; as well as A processing device coupled to the memory, the processing device being configured to access the memory, wherein, when executed by the processing device, the instructions cause the processing device to perform operations, the operations including: A request is received from a first user among a plurality of users to receive audio associated with a metaverse location in the virtual metaverse, wherein the first user is associated with a user device, and wherein the plurality of users are associated with a corresponding avatar among a plurality of avatars in the metaverse location; Obtain a data model associated with the metaverse site, wherein the data model includes one or more spatial parameters representing one or more physical laws applicable to the metaverse site; Avatar information and scene information are extracted from the data model, wherein the avatar information includes one or more of the following: the position, velocity, or direction of the plurality of avatars in the metaverse location, the plurality of avatars including the first avatar associated with the first user, and wherein the scene information includes one or more of the following: occlusions, echoes, or virtual walls that are virtually close to the first avatar in the metaverse location; Based on the avatar information and the scene information, the corresponding audio stream received from each of the plurality of users is transformed, and based on the one or more spatial parameters, one or more audio features of at least one of the corresponding audio streams are transformed to create a spatialized audio stream; Determine a set of prioritized audio streams received from each of the plurality of users, wherein transforming the corresponding audio streams further includes transforming the set of prioritized audio streams to create the spatialized audio stream, wherein audio streams associated with an avatar moving toward a receiving avatar are prioritized over audio streams associated with an avatar moving away from the receiving avatar. Combine the spatialized audio streams to create a combined spatialized audio stream; and The combined spatialized audio stream is provided to the user equipment.

13. The system according to claim 12, wherein, The spatial parameters include distance attenuation parameters used to attenuate audio based on the distance between avatars.

14. The system according to claim 12, wherein, The corresponding audio stream received from each of the plurality of users includes mono audio received by a microphone device, and wherein the combined spatialized audio stream includes stereo audio.

15. The system according to claim 14, wherein, The combined spatialized audio stream includes stereo audio generated by positioning each user's mono audio at the location of the corresponding user's avatar.

16. The system according to claim 12, wherein, The combined spatialized audio stream includes spatial audio and background audio, the spatial audio being based on the audio stream received from one of the plurality of users other than the first user, wherein the background audio is generated based on one or more of the following: Audio received from users other than the first user; and Audio generated based on the movement of the avatar within the metaverse space.

17. The system according to claim 12, wherein, The operation also includes: Based on the velocity of the avatar in the metaverse, the audio streams received from each of the plurality of users are prioritized.

18. The system according to claim 12, wherein, Determining the set of prioritized audio streams includes: The audio streams received from each of the plurality of users are prioritized based on one or more of the following: proximity of the avatar in the metaverse, orientation of the avatar in the metaverse, virtual objects in the metaverse that are close to the avatar, the capabilities of the user device, or the user preferences of the first user.

19. The system according to claim 18, wherein, The audio stream associated with the avatar closer to the receiving avatar takes precedence over the audio stream associated with the avatar farther from the receiving avatar, and wherein the audio stream associated with the avatar facing the receiving avatar takes precedence over the audio stream associated with the avatar facing away from the receiving avatar.

20. The system of claim 12, further comprising: A spatial audio manager for transforming the corresponding audio stream received from each of the plurality of users; as well as An audio device overlay module is configured to disable non-spatialized audio at the user equipment before providing the combined spatialized audio stream to the user equipment.

Images