Voice chat translation

Abstract

The implementations described herein relate to a method, system, and computer-readable medium for providing automatic translation of voice chat in virtual experiences. The automatic translation may hold context data and / or sentiment data extracted from input speech received from a first user. The context data and / or sentiment data may be used to translate the input speech into a second language for output to a second user at a user device.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application is an international application that claims the benefit of priority under 35 U.S.C. § 119(e) of U.S. Provisional Application No. 63 / 350,154, filed on June 8, 2022, entitled "VOICE CHAT TRANSLATION", the entire content of which is incorporated herein by reference.

[0002] Embodiments generally relate to audio input and output via a computer device, and more particularly, to methods, systems, and computer - readable media for providing voice chat translation that preserves the characteristics, context, and emotion of a user's voice in a virtual environment such as a metaverse place in a virtual metaverse

Background Art

[0003] Computer audio (e.g., chat between users of a computer device) often consists of providing monoral or stereo audio when it is received from a listening device or microphone. When the audio should be translated for various users speaking different languages, most solutions rely on text - based translation that provides only a simple function including literal or phrase - by - phrase translation presented in text. Thus, much of the context and / or emotion associated with a user's directed chat may be lost in translation.

[0004] The description of the background art provided herein is for the purpose of presenting the context of the present disclosure. Neither the research of the inventors named herein within the scope described in this background art section nor aspects of the description that may not be considered prior art at the time of filing are admitted as prior art to the present disclosure, either explicitly or implicitly.

Summary of the Invention

[0005] The implementation of this application relates to providing voice chat translation in a virtual metaverse.

[0006] According to one embodiment, a method implemented by a computer includes the steps of: receiving a request to translate audio related to a chat function of a metaverse place of a virtual metaverse, wherein the audio is received from a first user of a plurality of users, and the plurality of users are associated with a metaverse place; retrieving translation data related to a second user of a plurality of users, wherein the translation data includes at least language preferences related to the second user, and the second user is associated with a user device; converting the audio received from the first user into text, wherein the audio includes input speech in a first language spoken by the first user; translating the text into a second language, wherein the second language is defined by language preferences, and the translated text includes context data from the audio; converting the translated text into output speech including the context data; and providing the output speech to a user device.

[0007] In some implementations, the request identifies a first user and a second user, and the request originates from a computing device associated with the first user.

[0008] In some implementations, the method further includes the step of retrieving voice output preferences related to a first user, wherein the voice output preferences at least partially override translation data related to a second user.

[0009] In some implementations, the method further includes the step of moderating the text to remove words based on a text moderation filter before translating the text into a second language.

[0010] In some implementations, the translation step involves providing text as input to a trained machine learning model and receiving the translated text as output from the trained machine learning model.

[0011] In some implementations, contextual data includes sentiment data extracted from audio.

[0012] In some implementations, the method further includes a step of preprocessing the audio to extract sentiment data.

[0013] In some implementations, the step of converting translated text into output speech involves using a speech waveform modulator to create a moderated speech waveform that includes at least some contextual data.

[0014] In some implementations, the method further includes the steps of translating text into multiple different languages ​​to create multiple different translated texts, and converting the multiple different translated texts into multiple different output speeches.

[0015] In some implementations, the method includes the step of providing multiple different output speeches to multiple user devices, each output speech comprising utterances in the language associated with each of the multiple user devices.

[0016] In another example, a non-temporary computer-readable medium is disclosed that stores instructions causing the processing device to perform actions including receiving a request to translate audio related to the chat function of a metaverse place in a virtual metaverse, the audio being received from a first user of a plurality of users and associated with a metaverse place; retrieving translation data related to a second user of a plurality of users, the translation data including language preferences related to at least the second user and associated with a user device; converting the audio received from the first user into text, the audio including input speech in a first language spoken by the first user; translating the text into a second language, the second language defined by language preferences and the translated text including context data from the audio; converting the translated text into output speech including context data; and providing the output speech to the user device.

[0017] In some implementations, the request identifies a first user and a second user, and the request originates from a computing device associated with the first user.

[0018] In some implementations, the operation involves retrieving voice output preferences related to a first user, the voice output preferences, and further including retrieving such preferences that at least partially override translation data related to a second user.

[0019] In some implementations, the behavior further includes moderating the text to remove words based on a text moderation filter before translating the text into a second language.

[0020] In some implementations, translation involves providing text as input to a trained machine learning model and receiving the translated text as output from the trained machine learning model.

[0021] In some implementations, contextual data includes sentiment data extracted from audio.

[0022] In some implementations, the operation further includes preprocessing the audio to extract sentiment data.

[0023] In some implementations, converting translated text to output speech involves using a speech waveform modulator to create a modified speech waveform that includes, at least partially, contextual data.

[0024] In some implementations, the operation further includes translating text into multiple different languages ​​to create multiple different translated texts, converting the multiple different translated texts into multiple different output speeches, and providing the multiple different output speeches to multiple user devices, wherein each output speech includes utterances in the language associated with each of the multiple user devices.

[0025] According to yet another aspect, there is provided a system including a memory storing instructions and a processing device coupled to the memory and operable to access the memory, wherein the instructions, when executed by the processing device, cause the processing device to receive a request to translate audio related to a chat function of a metaverse place in a virtual metaverse, the audio being received from a first user among a plurality of users, the plurality of users being associated with the metaverse place; retrieve translation data related to a second user among the plurality of users, the translation data including at least a language preference related to the second user, the second user being associated with a user device; convert the audio received from the first user into text, the audio including input speech in a first language spoken by the first user; translate the text into a second language, the second language being defined by the language preference; translate the translated text into output speech including the context data from the audio; and provide the output speech to the user device.

[0026] In another aspect, details of some, features, and implementations of the systems, devices, methods, and non-transitory computer-readable media disclosed herein may be combined to form additional aspects that include omitting and / or modifying some or a portion of the individual components or features, including additional components or features and / or other modifications, and all such modifications are within the scope of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] [Figure 1] FIG. 12 is a diagram of an exemplary network environment for providing voice chat translation in a virtual metaverse according to some implementations. [Figure 2]A diagram of an exemplary network environment for providing voice chat translation in a virtual metaverse according to some implementations. [Figure 3] A diagram of an exemplary voice translation pipeline according to some implementations. [Figure 4A] A diagram showing an exemplary per-user voice machine learning model training method according to some implementations. [Figure 4B] A diagram showing an exemplary method for moderation and adjustment of voice chat according to some implementations. [Figure 4C] A diagram showing an exemplary method for player control of voice chat output according to some implementations. [Figure 4D] A diagram showing an exemplary voice generation method according to some implementations. [Figure 5] A diagram showing an exemplary voice machine learning model training method according to some implementations. [Figure 6] A flowchart of an exemplary method for providing voice chat translation in a virtual metaverse according to some implementations. [Figure 7] A flowchart of a further exemplary method for providing voice chat translation in a virtual metaverse according to some implementations. [Figure 8] A block diagram showing an exemplary computing device that may be used to implement one or more features described herein according to some implementations.

Modes for Carrying Out the Invention

[0028] One or more implementations described herein relate to voice chat translation related to an online virtual experience platform. Features may include, in a metaverse place of a virtual metaverse, automatically converting speech to text while preserving context and / or sentiment data, translating the text into different languages, and automatically generating speech from the translated text using context and / or sentiment data. The generated speech preserves at least some of the context and / or sentiment from the source speech.

[0029] The features described herein provide, for example, automatic translation of audio for output on client devices coupled to an online platform such as an online virtual experience platform or an online game platform. The online platform may provide a virtual metaverse having multiple associated metaverse places. A virtual avatar associated with a user can move between metaverse places, participate in metaverse places, and interact with items, characters, other avatars, and objects within the metaverse places. The avatar can move from one metaverse place to another while engaging in voice chat, which provides an immersive and enjoyable experience by enabling communication with users who speak different languages. Different audio streams from multiple users (avatars associated with multiple users) can be translated based on language preferences defined by the users and provided to other users.

[0030] Contextual and / or emotionally resonant automatic translation allows users to accurately understand the context and / or emotions conveyed through chat, despite language barriers. This can provide users of virtual experience platforms with a more immersive and enjoyable experience.

[0031] Online virtual experience platforms and online game platforms (also known as “user-generated content platforms” or “user-generated content systems”) provide various ways for users to interact with each other. For example, users of an online virtual experience platform may create games or other content or resources within the online platform, such as characters, graphics, and items for use in gameplay and / or within the virtual metaverse.

[0032] Users of online virtual experience platforms may collaborate towards common goals in metaverse places, games, or game creation, share various virtual items (e.g., inventory items, game items, etc.), engage in audio chat (e.g., audio chat with automatic translation), and send electronic messages to each other. Users of online virtual experience platforms may interact with other users, for example, by playing games that include characters (avatars) or other game objects and mechanisms. Online virtual experience platforms may also enable users of the platform to communicate with each other. For example, users of online virtual experience platforms may communicate with each other using voice messages or live voice interaction (e.g., voice chat with automatic translation), text messaging, video messaging (e.g., with audio translation), or a combination of the above. Some online virtual experience platforms may provide virtual 3D environments, or multiple linked environments within a metaverse, within which users can interact with each other or play online games.

[0033] To enhance the entertainment value of an online virtual experience platform, the platform may provide rich audio for playback on the user's device. This audio may include, for example, different audio streams from different users and background audio. According to various implementations described herein, different audio streams may be automatically translated based on the user who is capturing and listening. For example, a first user may request to engage in a voice chat with a second user with automatic translation. The audio stream from the first user may then be translated before being provided to the second user. Furthermore, audio streams may also be provided to other users, with or without automatic translation, based on, for example, user preferences, language preferences, override preferences, and / or other settings.

[0034] Figures 1-3: Exemplary system architecture Figure 1 shows an exemplary network environment 100 based on a partial implementation of the present disclosure. The network environment 100 (also referred to herein as the “System”) includes an online virtual experience platform 102, a first client device 110, a second client device 116 (collectively referred herein as “Client Devices 110 / 116”), all connected via a network 122. The online virtual experience platform 102 may include, among other things, a virtual experience engine 104, one or more virtual experiences 105, a voice chat translation component 106, and a data store 108.

[0035] Client device 110 may include a virtual experience application 112, and client device 116 may include a virtual experience application 118. Users 114 and 120 can use client devices 110 and 116, respectively, to interact with the online virtual experience platform 102 and other users utilizing the online virtual experience platform 102.

[0036] The network environment 100 is provided for illustrative purposes. In some implementations, the network environment 100 may include the same, fewer, more, or different elements, configured in the same or different ways as shown in Figure 1.

[0037] In some implementations, network 122 may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or wide area network (WAN)), a wired network (e.g., an Ethernet network), a wireless network (e.g., an 802.11 network, a Wi-Fi® network, or a wireless LAN (WLAN)), a cellular network (e.g., a Long-Term Evolution (LTE) network), a router, a hub, a switch, a server computer, or a combination thereof.

[0038] In some implementations, the datastore 108 may be non-temporary computer-readable memory (e.g., random-access memory), a cache, a drive (e.g., a hard drive), a flash drive, a database system, or another type of component or device capable of storing data. The datastore 108 may also include multiple storage components (e.g., multiple drives or multiple databases) that may span multiple computing devices (e.g., multiple server computers).

[0039] In some implementations, the online virtual experience platform 102 may include servers having one or more computing devices (e.g., cloud computing systems, rack-mount servers, server computers, clusters of physical servers, virtual servers, etc.). In some implementations, the servers may be included in the online virtual experience platform 102, be a separate system, or be part of another system or platform.

[0040] In some implementations, the online virtual experience platform 102 may include one or more computing devices (such as rack-mount servers, router computers, server computers, personal computers, mainframe computers, laptop computers, tablet computers, and desktop computers), data stores (e.g., hard disks, memory, databases), networks, software components, and / or hardware components that run on the online virtual experience platform 102 and are used to provide users with access to the online virtual experience platform 102. The online virtual experience platform 102 may also include a website (e.g., one or more web pages) or application backend software that is used to provide users with access to content provided by the online virtual experience platform 102. For example, users 114 / 120 may access the online virtual experience platform 102 using virtual experience applications 112 / 118 on client devices 110 / 116, respectively.

[0041] In some implementations, the online virtual experience platform 102 may include some kind of social network that provides connections between users, or some kind of user-generated content system that enables users (e.g., end users or consumers) to communicate with other users through the online virtual experience platform 102, and the communication may include voice chat (e.g., synchronous and / or asynchronous voice communication with or without automatic translation), video chat (e.g., synchronous and / or asynchronous video communication with or without automatic audio translation), or text chat (e.g., synchronous and / or asynchronous text-based communication with or without automatic text translation).

[0042] In some implementations of this disclosure, “User” may represent a single individual. However, other implementations of this disclosure may include the fact that “User” (e.g., a creator user) is an entity controlled by a set of users or an automated source. For example, a set of individual users united as a community or group within a user-generated content system may be considered “User.”

[0043] In some implementations, the online virtual experience platform 102 may be a virtual game platform. For example, the game platform may provide single-player or multiplayer games to a community of users who may access or interact with games (e.g., user-generated games or other games) using client devices 110 / 116 via the network 122. In some implementations, games (also referred to herein as “video games,” “online games,” “metaverse places,” or “virtual experiences”) may be, for example, two-dimensional (2D) games, three-dimensional (3D) games (e.g., 3D user-generated games), virtual reality (VR) games, or augmented reality (AR) games. In some implementations, users may search for games and game items and participate in gameplay with other users in one or more games. In some implementations, games may be played in real time with other users of the game. Similarly, some users may engage in real-time voice or video chat with other users of the game. As described herein, real-time voice or video chat may include automatic translation.

[0044] In some implementations, other collaboration platforms may be used in place of, or in addition to, the online virtual experience platform 102 and / or the voice chat translation component 106, along with the features described herein. For example, social networking platforms, purchasing platforms, messaging platforms, and creation platforms may be used with the automatic translation feature so that the translated audio is provided to users outside the game and / or virtual experience.

[0045] In some implementations, gameplay may refer to the interaction of one or more players using client devices (e.g., 110 and / or 116) within a game (e.g., virtual experience 105), or the representation of the interaction on the display or other output device of client device 110 or 116. In some implementations, gameplay may instead refer to the interaction within a virtual experience or metaverse place, which may be similar to, different from, or have the same purpose as some games. Furthermore, although referred to as “player,” the terms “avatar,” “user,” and / or other terms may be used to refer to users who engage with and / or interact with an online virtual experience.

[0046] One or more virtual experiences 105 are provided by an online virtual experience platform. In some implementations, a virtual experience 105 may include electronic files that can be executed or loaded using software, firmware, or hardware configured to present virtual content (e.g., digital media items) to entities. In some implementations, a virtual experience application 112 / 118 may run in conjunction with a virtual experience engine 104, and a virtual experience 105 may be rendered in conjunction with the virtual experience engine 104. In some implementations, a virtual experience 105 may have a common set of rules or common goals, and the virtual environment of a virtual experience 105 shares a common set of rules or common goals. In some implementations, different virtual experiences may have different rules or goals from one another.

[0047] In some implementations, a game and / or virtual experience may have one or more environments (also referred to herein as “game environments,” “metaverse places,” or “virtual environments”), and multiple environments may be linked. An example of an environment may be a three-dimensional (3D) environment. Virtual experience 105 or one or more environments of a virtual experience may collectively be referred herein as a “world,” “game world,” “virtual world,” “universe,” or “metaverse.” An example of a world may be a 3D metaverse place of virtual experience 105. For example, a user may construct a metaverse place that is linked to another metaverse place created by a different user than the first user. A character in a virtual experience may cross a virtual boundary to enter an adjacent metaverse place. Furthermore, sounds, theme music, and / or background music may also cross a virtual boundary so that an avatar standing near a virtual boundary may hear audio that includes at least some of the sounds emanating from an adjacent metaverse place.

[0048] It should be noted that 3D environments or 3D worlds use graphics that utilize a three-dimensional representation of geometric data representing the content (or at least present the content in a way that makes it appear as 3D content, regardless of whether a three-dimensional representation of geometric data is used). 2D environments or 2D worlds use graphics that utilize a two-dimensional representation of geometric data representing the game content.

[0049] In some implementations, the online virtual experience platform 102 may host one or more virtual experiences 105 and allow users to interact with the virtual experiences 105 using virtual experience applications 112 / 118 on client devices 110 / 116 (for example, to search for experiences, games, game-related content, virtual content, or other content). Users of the online virtual experience platform 102 (for example, 114 and / or 120) may play, create, interact with, or build a virtual experience 105, search for a virtual experience 105, communicate with other users, create and build objects for the virtual experience 105 (for example, also referred to herein as “items,” “game objects,” or “virtual game items”), and / or search for objects. For example, when creating user-generated virtual items, users may, among other things, create characters, decorations for characters, one or more virtual environments for an interactive game, or build structures used within the virtual experience 105.

[0050] In some implementations, users may buy, sell, or trade virtual game objects, such as in-platform currency (e.g., virtual currency), with other users of the online virtual experience platform 102. In some implementations, the online virtual experience platform 102 may transmit game content to a game application (e.g., the virtual experience application is 112). In some implementations, game content (also referred to herein as “Content”) may refer to any data or software instructions related to the online virtual experience platform 102 or the game application (e.g., game objects, games, user information, videos, images, commands, media items, etc.).

[0051] In some implementations, a game object (for example, also referred to herein as “item,” “object,” or “virtual game item”) may refer to an object used, created, shared, or otherwise depicted in a virtual experience 105 of an online virtual experience platform 102 or in a virtual experience application 112 or 118 of a client device 110 / 116. For example, a game object may include parts, models, characters, tools, weapons, clothing, buildings, vehicles, currency, flora, fauna, and components of the above (for example, windows of a building).

[0052] It should be noted that the online virtual experience platform 102 hosting the virtual experience 105 is provided for illustrative purposes only, not as an extension. In some implementations, the online virtual experience platform 102 may host one or more media items that may contain communication messages from one user to one or more other users. Media items may include, but are not limited to, digital video, digital movies, digital photographs, digital music, audio content, melodies, website content, social media updates, ebooks, e-magazines, digital newspapers, digital audiobooks, e-journals, web blogs, real simple syndication (RSS) feeds, e-comics, and software applications. In some implementations, media items may be electronic files that can be executed or loaded using software, firmware, or hardware configured to present digital media items to entities.

[0053] In some implementations, a virtual experience 105 may be associated with a specific user or a specific group of users (for example, a private game), or it may be made available to users of the online virtual experience platform 102 (for example, a public game). In some implementations where the online virtual experience platform 102 associates one or more virtual experiences 105 with a specific user or group of users, the online virtual experience platform 102 may associate a specific user with a virtual experience 105 using user account information (for example, a user account identifier such as a username and password). Similarly, in some implementations, the online virtual experience platform 102 may associate a specific developer or group of developers with a virtual experience 105 using developer account information (for example, a developer account identifier such as a username and password).

[0054] In some implementations, the online virtual experience platform 102 or client devices 110 / 116 may include a virtual experience engine 104 or a virtual experience application 112 / 118. The virtual experience engine 104 may include a virtual experience application similar to the virtual experience applications 112 / 118. In some implementations, the virtual experience engine 104 may be used for deploying or running a virtual experience 105. For example, the virtual experience engine 104 may include, among other features, a rendering engine ("renderer") for 2D, 3D, VR, or AR graphics, a physics engine, a collision detection engine (and collision response), a sound engine, a machine learning model, a translation component, a spatialized audio manager / engine, an audio mixer, an audio subscription exchange, audio subscription logic, an audio subscription prioritizer, a real-time communication engine, scripting capabilities, an animation engine, an artificial intelligence engine, networking capabilities, streaming capabilities, memory management capabilities, threading capabilities, scene graph capabilities, or video support for cinematic techniques. The components of the virtual experience engine 104 may generate commands (e.g., rendering commands, collision commands, physics commands, etc.) to help compute and render the virtual experience, and may translate audio (e.g., convert audio to text, translate text, convert translated text to speech, etc.). In some implementations, the virtual experience applications 112 / 118 of client devices 110 / 116 may work independently, in cooperation with the virtual experience engine 104 of the online virtual experience platform 102, or in a combination of both.

[0055] In some implementations, both the online virtual experience platform 102 and client devices 110 / 116 run virtual experience engines (104, 112, and 118, respectively). The online virtual experience platform 102, using virtual experience engine 104, may execute some or all of the functions of the virtual experience engine (e.g., generate physical commands, rendering commands, spatialized audio commands, etc.) or offload some or all of the functions of the virtual experience engine to the virtual experience engine 104 of client device 110. In some implementations, each virtual experience 105 may have different ratios between the functions of the virtual experience engine executed on the online virtual experience platform 102 and the functions of the virtual experience engine executed on client devices 110 and 116.

[0056] For example, the virtual experience engine 104 of the online virtual experience platform 102 may be used to generate physical commands when collisions exist between at least two virtual objects, while further virtual experience engine functions (e.g., generating rendering commands or combining spatialized audio streams) may be offloaded to the client device 110. In some implementations, the ratio of virtual experience engine functions executed on the online virtual experience platform 102 to those executed on the client device 110 may be changed (e.g., dynamically) based on gameplay conditions. For example, if the number of users participating in the gameplay of virtual experience 105 exceeds a threshold number, the online virtual experience platform 102 may execute one or more virtual experience engine functions previously executed by the client device 110 or 116.

[0057] For example, a user may interact with the virtual experience 105 on client devices 110 and 116 and send control commands to the online virtual experience platform 102 (e.g., user input such as right, left, up, down, user selection, or character position and velocity information). After receiving control commands from client devices 110 and 116, the online virtual experience platform 102 may send gameplay commands to client devices 110 and 116 based on the control commands (e.g., position and velocity information for characters participating in group gameplay, or commands such as rendering commands, collision commands, spatialized audio commands). For example, the online virtual experience platform 102 may perform one or more logical actions according to the control commands (e.g., using the virtual experience engine 104) to generate gameplay commands for client devices 110 and 116. Otherwise, the online virtual experience platform 102 may pass one or more control commands from one client device 110 to other client devices (e.g., 116) participating in the virtual experience 105. Client devices 110 and 116 may use gameplay commands to render gameplay for presentation on the displays of client devices 110 and 116.

[0058] In some implementations, control commands may refer to commands that indicate an in-experience action for the user's character. For example, control commands may include user input, user selection, gyroscope position and orientation data, force sensor data, etc., to control in-experience actions such as right, left, up, down. Control commands may also include character position and velocity information. In some implementations, control commands are sent directly to the online virtual experience platform 102. In other implementations, control commands may be sent from client device 110 to another client device (e.g., 116), which then generates gameplay commands using the local virtual experience engine 104. Control commands may include commands to play voice communication messages or other sounds from another user through an audio device (e.g., speakers, headphones, etc.).

[0059] In some implementations, gameplay instructions may refer to instructions that enable client device 110 (or 116) to render gameplay for a virtual experience, such as a multiplayer game or virtual experience. Gameplay instructions may include one or more of the following: user input (e.g., control instructions), character position and velocity information, or commands (e.g., physics commands, rendering commands, collision commands, etc.).

[0060] In some implementations, a character (or more broadly, a virtual object) is constructed from components that automatically combine to assist the user in editing, and one or more of these components may be selected by the user. One or more characters (also referred herein as “avatars” or “models”) may be associated with a user, and the user may control the characters to facilitate the user’s interaction with Game 105. In some implementations, a character may include components such as body parts (e.g., head, hair, arms, legs, etc.) and accessories (e.g., T-shirts, glasses, decorative images, tools, etc.). In some implementations, customizable character body parts may include, among other things, head type, body part type (arms, legs, torso, and hands), face type, hair type, and skin type. In some implementations, customizable accessories may include clothing (e.g., shirts, trousers, hats, shoes, glasses, etc.), weapons, or other tools.

[0061] In some implementations, users may also control the character's scale (e.g., height, width, or depth) or the scale of the character's components. In some implementations, users may control the character's proportions (e.g., blocky, anatomical). In some implementations, a character may not include character game objects (e.g., body parts), but it should be noted that users may control the character (without character game objects) to facilitate user interaction with the virtual experience (for example, a puzzle game where there are no rendered character game objects, but the user still controls the character to control in-game actions).

[0062] In some implementations, components such as body parts may be basic geometric shapes such as blocks, cylinders, spheres, or some other basic shapes such as wedges, rings, tubes, or channels. In some implementations, the creator module may expose the user's character for viewing or use by other users of the online virtual experience platform 102. In some implementations, creating, modifying, or customizing characters, other virtual objects, virtual experiences 105, or virtual environments may be done by the user using a user interface (e.g., a developer interface), with or without scripting (or with or without an application programming interface (API)). It should be noted that, for illustrative purposes only and not limitation, characters are described as having the form of a humanoid robot. It should be further noted that characters may have any form, such as a vehicle, an animal, an inanimate object, or other creative form.

[0063] In some implementations, the online virtual experience platform 102 may store user-created characters in a data store 108. In some implementations, the online virtual experience platform 102 maintains a character catalog and a virtual experience catalog that may be presented to the user via the virtual experience engine 104, the virtual experience 105, and / or client devices 110 / 116. In some implementations, the virtual experience catalog includes images of virtual experiences stored in the online virtual experience platform 102. In addition, the user may select a character (for example, a character created by the user or another user) from the character catalog to participate in a selected experience. The character catalog includes images of characters stored in the online virtual experience platform 102. In some implementations, one or more characters in the character catalog may be created or customized by the user. In some implementations, the selected character may have character settings that define one or more of the character's components.

[0064] In some implementations, a user's character may include component configurations, and the configuration and appearance of these components, as well as the character's appearance more broadly, may be defined by character settings. In some implementations, the user's character settings may be selected by the user, at least partially. In other implementations, the user may select a character with default character settings or other user-selected character settings. For example, the user may select a default character from a character catalog that has predefined character settings, and further, the user may customize the default character by changing some of the character settings (for example, by adding a shirt with a customized logo). Character settings may be associated with a specific character by the online virtual experience platform 102.

[0065] In some implementations, client devices 110 or 116 may include, respectively, computing devices such as personal computers (PCs), mobile devices (e.g., laptops, mobile phones, smartphones, tablet computers, or netbooks), network-connected televisions, and game consoles. In some implementations, client devices 110 or 116 may also be referred to as “user devices.” In some implementations, one or more client devices 110 or 116 may connect to the online virtual experience platform 102 at any time. It should be noted that the number of client devices 110 or 116 is given as an example, not an exemption. In some implementations, any number of client devices 110 or 116 may be used.

[0066] In some implementations, each client device 110 or 116 may contain an instance of the virtual experience application 112 or 118. In one implementation, the virtual experience application 112 or 118 may enable the user to use and interact with the online virtual experience platform 102, such as searching for virtual experiences, virtual items, or other content, controlling virtual characters in virtual experiences hosted by the online virtual experience platform 102, or viewing or uploading content such as virtual experiences 105, images, video items, web pages, or documents. In one example, the virtual experience application may be a web application (e.g., an application that works in conjunction with a web browser) that can access, retrieve, present, or navigate content provided by a web server (e.g., virtual characters in a virtual environment). In another example, the virtual experience application may be a native application (e.g., a mobile application, app, or game program) that is installed and runs locally on a client device 110 or 116 and enables the user to interact with the online virtual experience platform 102. The virtual experience application may render, display, or present content (e.g., web pages, user interfaces, media viewers, audio streams) to the user. In an implementation, the virtual experience application may also include an embedded media player embedded in a web page.

[0067] In aspects of this disclosure, the virtual experience application 112 / 118 may be an online virtual experience platform application for users to build, create, and edit content, upload it to the online virtual experience platform 102, and interact with the online virtual experience platform 102 (for example, to play a virtual experience 105 hosted by the online virtual experience platform 102). Accordingly, the virtual experience application 112 / 118 may be provided to client devices 110 or 116 by the online virtual experience platform 102. In another example, the virtual experience application 112 / 118 may be an application downloaded from a server.

[0068] In some implementations, users may log in to the online virtual experience platform 102 via a virtual experience application. Users may access their user account by providing user account information (e.g., username and password), and the user account is associated with one or more characters available to participate in one or more virtual experiences 105 on the online virtual experience platform 102.

[0069] In general, the functions described as being performed by the online virtual experience platform 102 may also be performed by client devices 110 or 116 or a server in other implementations, as appropriate. In addition, functions attributed to specific components may be performed by different or multiple components working together. Furthermore, the online virtual experience platform 102 can be accessed as a service provided to other systems or devices through appropriate application programming interfaces (APIs), and is therefore not limited to use on websites.

[0070] In some implementations, the online virtual experience platform 102 may include a voice chat translation component 106.

[0071] In some implementations, the voice chat translation component 106 may include an Application Programming Interface (API) which contains a set of computer-executable code that provides functionality to the user and / or developer in the form of function calls that enable the software component to communicate and / or provide / receive data. The API may include several defined software functions related to voice chat translation, which can be used by the developer to enable audio translation functionality for voice chat and video chat, and may include any functions related to audio playback on the user device.

[0072] In some implementations, the voice chat translation component 106 is a software component that provides automatic voice chat translation functionality based on user settings. For example, in some implementations, the voice chat translation component may include one or more machine learning models, one or more text translation components, one or more audio conversion components, one or more text-to-speech components, one or more plugins for communicating with multiple third-party services, and / or any other suitable components. Figures 3 and 4 show different subcomponents that may be included as part of the voice chat translation component 106 in some implementations.

[0073] The operation of the online virtual experience platform 102 in providing automatic voice chat translation will be explained in more detail below with reference to Figure 2.

[0074] Figure 2 is a diagram of an exemplary network environment 200 (e.g., a subset of network environment 100) for providing automatic voice chat translation in a virtual metaverse, as implemented by some implementations. Network environment 200 is provided for illustrative purposes only. In some implementations, network environment 200 may contain the same, fewer, more, or different elements configured in the same or different ways as shown in Figure 2.

[0075] As shown in Figure 2, the online virtual experience platform 102 may communicate with client devices 110 and 116 via the network 122 so that a user audio stream 232 is received from client device 110 and a translated audio stream 234 is provided for output at client device 116. The online virtual experience platform 102 may also communicate with communication server 202 and relay server 210 via the network 122.

[0076] The online virtual experience platform 102 may include, in addition to the components shown in Figure 1, a voice chat plugin 208 for communicating with the communication server 202. The voice chat plugin 208 may perform the separation of audio streams and / or identify which audio streams are to be translated by the voice chat translation component 106. In this way, the audio stream 232 may be sent to any client device in its native form, but the voice chat plugin 208 may indicate to the media server 204 that a translated version of the audio stream 232 should be provided to other client devices. Thus, the voice chat plugin 208 may allow both the native communication and the translated communication to occur at substantially the same or similar time.

[0077] The communication server 202 may be a third-party communication server and / or a separate server located within the online virtual experience platform 102. The communication server 202 may include a media server 204 that communicates with the chat service 206 to enable operation.

[0078] The media server 204 is a server configured to connect components of the network environment 100 and transmit audio streams (or other data) between those components. The media server 204 may facilitate real-time communication, for example, between various client devices and between each client device and the online virtual experience server 102.

[0079] The chat service 206 may be a software service configured to enable voice chat and / or video chat (with audio) between a client device and the online virtual experience server 102.

[0080] The relay server 210 may be a third-party relay server and / or a separate server located on the online virtual experience platform 102. The relay server 210 may include a TURN server 212 that communicates with the TURN management component 214 in an operable manner.

[0081] The TURN server 212 may implement a traversal using relay NAT (TURN) protocol. The TURN server 212 may relay network traffic. For example, the TURN server 212 may support communication between client device 110 and client device 116 over network 122.

[0082] In addition to other functions, the TURN management component 214 may implement a communication protocol and control messaging with the TURN server 212.

[0083] Later, the voice chat translation component 106 and the translation of audio for chats using available translation data will be explained in more detail with reference to Figure 3.

[0084] Figure 3 is a diagram of an exemplary voice translation pipeline 300 for automatically translating voice chat (or video chat with audio) in a virtual metaverse, as in some implementations. Pipeline 300 is provided for illustrative purposes only. In some implementations, pipeline 300 may contain the same, fewer, more, or different elements configured in the same or different ways as shown in Figure 3.

[0085] As shown in Figure 3, pipeline 300 begins with receiving source audio from a voice chat (or video chat) in step 302. The source audio may be associated with translation data acquired in step 304. The translation data may include user settings for translation, language settings, and other user settings.

[0086] Upon acquiring translation data and receiving audio, the voice chat translation component 106 may begin translating (for example, as shown in the dotted box 306).

[0087] In step 308, the source audio may be converted from the format received from the media server 204 to another format suitable for text extraction. For example, if the media server 204 uses a first format (e.g., OPUS), step 308 may include conversion from the first format to a second format (e.g., WAV).

[0088] In step 310, the converted audio is (optionally) converted to text. For example, the converted audio may be processed to extract phonemes or other audio cues, and those phonemes or other audio cues may be used to estimate text. In some implementations, a trained machine learning model is used to convert the audio to text.

[0089] In step 312, machine translation of the text is performed to translate the text from the first language to the second language. The machine translation may retain context data and / or sentiment data. For example, context data and / or sentiment data may be identified using a trained machine learning model that identifies context and / or sentiment from phrases, phonemes, audio cues, accents, stresses, etc., in the received audio stream. In some implementations, context data may include specific stresses, accents, and other attributes from the first speaker. In these and other implementations, sentiment data may be extracted from context data (for example, by identifying stronger sentiments with stronger accents / stresses). In some implementations, a trained machine learning model or submodel may preprocess the audio to identify context data and / or sentiment data for use in the translated speech waveform (for example, by adjusting to enhance or suppress sentiment in the synthesized speech).

[0090] In some implementations, contextual and / or sentiment data may be further identified based on an analysis of the video or animation contained in the received data (when the chat is a video chat). Such analysis may be performed by a trained machine learning model or other technique configured to identify sentiment from one or more frames of the video or animation.

[0091] In step 314, the translated text is converted into speech using a speech synthesizer or TTS (text-to-speech). The speech synthesizer may utilize contextual and / or sentiment data (or directly in the generation process) to modify the generated speech waveform to output speech that conveys the same context and / or sentiment. For example, the speech synthesizer may receive input sentiment data and provide accented pronunciation in the output speech that reflects the sentiment indicated by the sentiment data. For example, the speech synthesizer may provide a fluctuating speech pattern that reflects the context indicated by the context data. For example, if the received audio is from an indoor context with reverberation or background noise, the output waveform may be generated to include the reverberation or background noise.

[0092] Other techniques to improve the translation of emotion and context may also be used. For example, speech-to-speech translation systems that make translations more expressive may be phoneme-based. Voices may be broken down into phonemes, with some differences between different languages ​​and their dialects. A probability matrix may be based on the speaker and the linguistic characteristics of their source language. Using this probability matrix, the most likely phoneme following other phonemes can be effectively rendered and / or probabilistically identified.

[0093] In step 316, the speech waveform is (optionally) converted back from the second format to the first format. In this way, the audio output step 318 may provide an audio stream that is input by the media server 204 and can be directed to chat recipients in a manner similar to untranslated voice chat.

[0094] Hereafter, a brief description of several exemplary methods for performing automatic voice chat translation will be given with reference to Figures 4A to 4D.

[0095] Figures 4A-4D: Exemplary methods of automatic voice translation Figure 4A shows an exemplary per-user voice machine learning model training method 400 in some implementations. As shown, method 400 may include training one or more per-user machine learning models. Method 400 may also include storing models that have been trained to provide improved translation accuracy and are associated with specific users. Multiple trained models may also be used for multilingual translation. Method 400 may use transfer learning techniques in some implementations.

[0096] As shown in Figure 4A, user 414 may provide input voice chat audio 402 to voice chat server 404 (or server 102). The voice processing plugin 406 and voice preprocessing stage 408 may filter and / or remove noise or other artifacts from the audio 402. The training data injection system 422 then generates training records for training a machine learning model and may store the training records in the training data store 434. In some implementations, a training data cleanup processor 438 may refurbish the stored training records.

[0097] Subsequently, or substantially simultaneously, the machine learning model evaluator processor 424 and the machine learning model generator processor 432 may generate a data model representing the machine learning model being trained and store the data model in the model data store 426. Similarly, different versions of the machine learning model may be stored in the data store 428. A reference model and / or base model may be stored and / or retrieved from the data store 436 for use in training to create the models stored in 426 / 428.

[0098] As shown in Figure 4A, in some implementations, machine learning models may be generated, trained, and tuned for each user. In this way, faster voice chat translation with improved context and / or emotion may be achieved. Other variations, including machine learning models based on specific dialects, specific languages, etc., may be implemented in some implementations instead of, or in combination with, a user-specific model.

[0099] Figure 4B shows an exemplary voice chat moderation and adjustment method 410 by some implementations. As shown, method 410 may include extending the voice translation system to include trust and safety features that enable any listener (e.g., a young listener for whom certain content may be inappropriate or unacceptable) to participate in the voice chat safely. Method 410 may filter inappropriate voice chat messages, block audio that does not contain voice, and allow a user to adjust how voice sounds to recipients of the voice chat.

[0100] As shown in Figure 4B, user 414 initiates a chat request to chat with user 416. Voice chat audio 402 is sent from the user device associated with user 414 to the voice chat server 404 (or server 102). The voice processing plugin 406 sends the processed audio waveform to the speech-to-text (STT) system 444 to create text. The text may then undergo text moderation and / or filtering 448 to remove offensive or moderated content.

[0101] In some implementations, the steps of the voice adjustment preprocessor 442 and voice adjustment 446 may be performed to adjust the synthesized speech waveform to mimic the emotion in the translated language. Furthermore, in some implementations where direct phoneme translation may be used, preprocessing 442 may also include moderation activities based on phonemes related to the moderated content.

[0102] Figure 4C shows an exemplary method 420 for player control of voice chat output in one implementation. As shown, method 420 may include a voice adjustment system 458 to allow a user to control how their voice sounds to other users.

[0103] As shown in Figure 4C, user 414 requests to chat with user 416. The voice chat audio 402 is provided to the voice chat server 404 (or server 102) as described above and undergoes voice processing 406. In some implementations, the voice output preferences 452 (i.e., including override preferences and other preferences related to user 414) are sent to the voice preference service 454 for storage in the data store 462.

[0104] The voice output generation system 456 may retrieve user-trained machine learning models from datastore 464 and / or voice pack models from datastore 466 for use in synthesizing speech waveforms from translated text. The voice adjustment system 458 may then create desired voice chat output audio to be sent to the voice chat server 404 and routed to user 416.

[0105] Figure 4D shows an exemplary voice generation method 430 by some implementations. As shown, method 430 may include a voice generation application programming interface (API) exposed to developers. Developers may use the exposed API to add voices to non-player characters (NPCs) using text input, and may also include localized voices based on a speech translation pipeline 300. For example, multilingual output may be provided for different texts entered by developers for output in different regions.

[0106] As shown in Figure 4D, native language information 470, NPC text information 472, and voice characteristic definitions 474 may be provided to platform 102, from which they are routed to audio data store 490, machine translation system 312, and text-to-speech (TTS) voice generation system 314. The machine translation system may generate multiple different translated texts 482 for generating multiple output speeches 486 in different languages ​​associated with the target computing device and / or user device, respectively. In addition, although shown as associated with NPCs, they may vary to include multiple translations of user chats so that multiple different recipients speaking different languages ​​may engage in the chat.

[0107] Figure 5 shows an exemplary voice machine learning model training method 500 with some implementations. As shown, method 500 may include training the machine learning model with greater reliability through backpropagation and the use of text-to-speech services.

[0108] For example, user audio 502 may be used by the STT component 508 to create text 510. The model management system 504 may use the backpropagated user audio 512 and the translated speech text 516 to perform a user-specific model training method 506 to improve accuracy.

[0109] Other variations of the exemplary method shown in Figure 5 are applicable to some implementations, and all such variations are considered to be within the scope of the exemplary embodiment.

[0110] Figure 6: An exemplary method for translating voice chat. Figure 6 is a flowchart of an exemplary method 600 for automatically translating voice chat in a metaverse place, as in some implementations. In some implementations, method 600 may be implemented, for example, on a server system, such as the online virtual experience platform 102 shown in Figure 1. In some implementations, some or all of method 600 may be implemented on a system such as one or more client devices 110 and 116 shown in Figure 1, and / or on both the server system and one or more client systems. In the examples described, the implementing system includes one or more processors or processing circuits and one or more storage devices such as a database or other accessible storage. In some implementations, one or more different components of the server and / or client may execute different blocks or other parts of method 600. Method 600 may begin in block 602.

[0111] In block 602, a request to translate audio is received. The audio is associated with a chat function in a metaverse place of the virtual metaverse from the first user among multiple users. The audio is received from the first user. Multiple users are associated with chatting in the metaverse place and / or with the first user. Furthermore, the first user may be associated with a first user device (such as client device 110). Block 602 is followed by block 604.

[0112] In block 604, translation data related to the second user among multiple users is retrieved. The translation data includes at least the language settings related to the second user. Furthermore, the second user is associated with a second user device (for example, client device 116). Block 604 is followed by block 606.

[0113] In block 606, audio received from the first user is converted to text. The audio includes input speech in a first language spoken by the first user. For example, a machine learning model may be trained to extract phonemes from the audio, use the extracted phonemes to reconstruct text related to the input speech, and extract context from the audio. Block 606 is followed by block 608.

[0114] In block 608, the text is translated into a second language. The second language is defined by language preferences, and the translated text includes contextual data and / or sentiment data. For example, a machine learning model may be used to extract contextual data based on the speech of the first user, and may also be used to extract sentiment data based on the speech of the first user. The contextual data and / or sentiment data may be encoded in any preferred format, including accent, variation, and other notations that may be embedded in the text and / or included separately from the text with appropriate timestamps or synchronization marks. Block 608 is followed by block 610.

[0115] In block 610, the translated text is converted into an output speech that includes context data and / or sentiment data. For example, a user-specific or user-perfect text-to-speech model may be provided with the text, context data, and / or sentiment data. The text-to-speech model may also be called a speech synthesizer or speech synthesis model. Using the context data and / or sentiment data together with the speech data, the speech synthesizer may generate a speech-based speech waveform in a second language that includes at least one or more of the context data and / or sentiment data. Block 610 is followed by block 612.

[0116] In block 612, the output speech is provided to a second user device. For example, the speech waveform may be converted to a specific audio format for routing using the relay server 210 and / or processing by the media server 204. The second user device (e.g., client device 116) may then receive and output the audio for playback to the second user.

[0117] As described above, systems, methods, and computer-readable media may provide automatic translation of voice chat in virtual experiences. Variations of the above technologies may include additional features that result in an improved user experience and reduced latency in translation.

[0118] For example, the following improvements may be implemented in method 600 in Figure 6 and / or pipeline 300 in Figure 3.

[0119] Different voice models: Each output model may be trained as a specific voice. Therefore, special language models may be implemented so that any user can speak as virtually any character voice available as an output model. Denoising / training for different age groups and dialects may generate different output models, which may be used to change the apparent age of the voice to suit the user or match user settings. Different dialects may also generate different output models, which may be used to modify the speech waveform to more closely match the regional dialect.

[0120] Safety: Because the expressiveness of voices can be tracked, the system may mute voice output when a player is speaking aggressively. Furthermore, speech-to-text (STT) may be applied to output voices to check for inappropriate content and / or context before providing audio to a second user device. Additionally, voice output may be modified to anonymize the original speaker's voice without losing the expressiveness of the voice.

[0121] Latency: Predictive models of phoneme mapping to other phonemes in other languages ​​can be used to reduce latency. For example, English has about 42 distinct phonemes, and Spanish has about 24, and probabilistic models that map phonemes as streams to their mappings in other languages ​​using segmentation based on audio data and translation can be used to reduce latency. Thus, the focus can be on how sounds and sounds are planned, so that latency inherent in word-to-word translation can be avoided in some scenarios. For example, a stream may be translated as it arrives, without waiting for the block or entire sentence to be fully uttered.

[0122] In these examples, low-latency voice translation may be possible for languages ​​with different structures. Take, for example, languages ​​with an adjective-noun-verb structure compared to verb-noun-adjective. Sounds are mapped probabilistically, potentially ignoring the structure of the sentence itself. To obtain reliable results, audio translation may be completed before the person finishes speaking.

[0123] In these examples, some structuring mismatches may still exist, but in many cases, they can be found early on before the audio stream ends. As a step towards this phoneme-by-phoneme processing, the final audio can be reconstructed during processing according to the linguistic rules of each language.

[0124] Figure 7: An exemplary method for translating voice chat based on phoneme prediction. Figure 7 is a flowchart of an exemplary method 700 for automatically translating voice chat in a metaverse place, as in some implementations. In some implementations, method 700 may be implemented, for example, on a server system, such as the online virtual experience platform 102 shown in Figure 1. In some implementations, some or all of method 700 may be implemented on a system such as one or more client devices 110 and 116 shown in Figure 1, and / or on both the server system and one or more client systems. In the examples described, the implementing system includes one or more processors or processing circuits and one or more storage devices such as a database or other accessible storage. In some implementations, one or more different components of the server and / or client may execute different blocks or other parts of method 700. Method 700 may begin in block 702.

[0125] In block 702, a request to translate audio is received. The audio is associated with a chat function in a metaverse place of the virtual metaverse from the first user among multiple users. The audio is received from the first user. Multiple users are associated with chatting in the metaverse place and / or with the first user. Furthermore, the first user may be associated with a first user device (such as client device 110). Block 702 is followed by block 704.

[0126] In block 704, translation data related to the second user among multiple users is retrieved. The translation data includes at least the language settings related to the second user. Furthermore, the second user is associated with a second user device (for example, client device 116). Block 704 is followed by block 706.

[0127] In block 706, audio received from the first user is converted into phonemes. The audio includes input speech in a first language spoken by the first user. For example, a machine learning model may be trained to extract phonemes from the audio and then use the extracted phonemes to further extract context from the audio. Block 706 is followed by block 708.

[0128] In block 708, phonemes from block 706 are processed phoneme-by-phoneme to determine a high-confidence match of phonemes between the first and second languages. For example, the phoneme-by-phoneme processing may include predictive phoneme processing based on probabilistically generated translated audio. The final results may be selected based on confidence levels, with higher confidence levels being selected first.

[0129] Phoneme-by-phoneme processing may also include reconstructing the audio based on probabilistically generated translated audio that varies based on the target language. In this way, the streaming structure can be adapted for different languages. Furthermore, the methods implemented in this manner by the computer may include predictive translation during speech-to-speech synthesis, translation during speech-to-speech synthesis, and other speech-to-speech synthesis methods. Block 708 is followed by Block 710.

[0130] In block 710, output speech, including context data and / or sentiment data, is generated based on highly reliable phoneme prediction. For example, a user-specific or user-specific speech model may be provided with phonemes, context data, and / or sentiment data. The speech model may also be called a speech synthesizer or speech synthesis model. Using the context data and / or sentiment data together with the phoneme data, the speech synthesizer may generate phoneme-based speech waveforms in a second language, including at least one or more of the context data and / or sentiment data. Block 710 is followed by block 712.

[0131] In block 712, the output speech is provided to a second user device. For example, the speech waveform may be converted to a specific audio format for routing using the relay server 210 and / or processing by the media server 204. The second user device (e.g., client device 116) may then receive and output the audio for playback to the second user.

[0132] A more detailed description of various computing devices that may be used to implement the different devices shown in Figures 1 to 6 is provided below with reference to Figure 8.

[0133] Figure 8: Exemplary Computing Device Figure 8 is a block diagram of an exemplary computing device 800 that may be used to implement one or more of the features described herein, in some implementations. In one example, device 800 may be used to implement a computer device (e.g., 102, 110, and / or 116 in Figure 1) and to perform an implementation of a suitable method described herein. Computing device 800 can be any suitable computer system, server, or other electronic or hardware device. For example, computing device 800 can be a mainframe computer, desktop computer, workstation, portable computer, or electronic device (portable device, mobile device, cell phone, smartphone, tablet computer, television, TV set-top box, personal digital assistant (PDA), media player, game device, wearable device, etc.). In some implementations, device 800 includes a processor 802, memory 804, an input / output (I / O) interface 806, and audio / video input / output devices 814 (e.g., a display screen, touchscreen, display goggles or glasses, audio speakers, headphones, microphone, etc.).

[0134] Processor 802 can be one or more processors and / or processing circuits for executing program code and controlling the basic operation of device 800. “Processor” includes any suitable hardware and / or software system, mechanism, or component for processing data, signals, or other information. A processor may include a general-purpose central processing unit (CPU), multiple processing units, a system with dedicated circuits for implementing functions, or other systems. Processing is not necessarily limited to a specific geographical location or subject to temporal constraints. For example, a processor may perform its functions in “real-time,” “offline,” “batch mode,” etc. Parts of the processing may be performed at different times and in different locations by different (or the same) processing systems. A computer may be any processor communicating with memory.

[0135] Memory 804 is generally located within device 800 for access by processor 802 and is suitable for storing instructions for execution by the processor. It may be any suitable processor-readable storage medium located separately from and / or integrated with processor 802, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc. Memory 804 can store software that runs on server device 800 by processor 802, including the operating system 808, application 810, and associated data 812. In some implementations, application 810 may include instructions that enable processor 802 to perform some or all of the functions described herein, for example, the methods shown in Figures 6 and / or 7.

[0136] For example, memory 804 may contain software instructions for automatically translating voice chat in the metaverse place. Any software in memory 804 may, alternatively, be stored in any other suitable storage location or computer-readable medium. Furthermore, memory 804 (and / or other connected storage devices) may store instructions and data used in the features described herein. Memory 804 and any other type of storage (magnetic disks, optical disks, magnetic tapes, or other tangible media) may be considered “storage” or “storage devices”.

[0137] The I / O interface 806 can provide functionality to enable the server device 800 to interface with other systems and devices. For example, network communication devices, storage devices (e.g., memory and / or datastore 108), and input / output devices can communicate via interface 806. In some implementations, the I / O interface may connect to interface devices including input devices (such as keyboards, pointing devices, touchscreens, microphones, cameras, scanners, etc.) and / or output devices (such as display devices, speaker devices, printers, monitors, etc.).

[0138] For ease of illustration, Figure 8 shows one block for each of the following: processor 802, memory 804, I / O interface 806, software blocks 808 and 810, and database 812. These blocks may represent one or more processors or processing circuits, operating systems, memory, I / O interfaces, applications, and / or software modules. In other implementations, device 800 may not have all of the components shown and / or may have other elements, including other types of elements, in place of or in addition to the elements shown herein. Although the online virtual experience platform 102 is described as performing the operation as described in some implementations herein, any suitable component or combination of components of the online virtual experience platform 102 or a similar system, or any suitable one or more processors associated with such a system, may perform the operation described.

[0139] A user device may implement and / or be used in conjunction with the features described herein. An exemplary user device may be a computer device including several components similar to device 800, for example, a processor 802, memory 804, and an I / O interface 806. A suitable operating system, software, and applications for the client device may be provided in memory and used by the processor. The I / O interface for the client device may be connected to network communication devices as well as input and output devices, for example, a microphone for capturing sound, a camera for capturing images or video, an audio speaker device for outputting sound, a display device for outputting images or video, or other output devices. For example, a display device in the audio / video input / output device 814 may be connected to (or included in) device 800 to display pre- and post-processing of images as described herein, and such a display device may include any suitable display device, for example, an LCD, LED, or plasma display screen, a CRT, a television, a monitor, a touchscreen, a 3-D display screen, a projector, or other visual display device. Some implementations may provide audio output devices, such as text-to-speech voice output or synthesis.

[0140] The methods, blocks, and / or operations described herein may be executed in a different order than shown or described, and / or concurrently (partially or completely) with other blocks or operations. Some blocks or operations may be executed for one part of the data and then executed again later, for example, for another part of the data. Not all described blocks and operations may be executed in all implementations. In some implementations, blocks and operations may be executed multiple times, in different orders, and / or at different times within the method.

[0141] In some implementations, some or all of the methods may be performed on a system such as one or more client devices. In some implementations, one or more of the methods described herein may be performed, for example, on a server system and / or on both a server system and a client system. In some implementations, one or more different components of a server and / or client may perform different blocks, operations, or other parts of the methods.

[0142] One or more methods described herein (for example, methods 400-700) can be implemented by computer program instructions or code that can be executed on a computer. For example, the code can be implemented by one or more digital processors (for example, microprocessors or other processing circuits) and can be stored in computer program products including non-temporary computer-readable media (for example, storage media), such as magnetic, optical, electromagnetic, or semiconductor storage media, including semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random-access memory (RAM), read-only memory (ROM), flash memory, hard magnetic disks, optical disks, solid-state memory drives, etc. Program instructions can also be contained in electronic signals and provided as electronic signals, for example, in the form of software as a service (SaaS) delivered from a server (for example, a distributed system and / or a cloud computing system). Alternatively, one or more methods can be implemented in hardware (such as logic gates) or in a combination of hardware and software. Exemplary hardware can include programmable processors (e.g., field-programmable gate arrays (FPGAs), complex programmable logic devices), general-purpose processors, graphics processors, and application-specific integrated circuits (ASICs). One or more methods may run as part of or as a component of an application running on the system, or as an application or software running in conjunction with other applications and operating systems.

[0143] One or more methods described herein may be performed as standalone programs that can run on any type of computing device, programs that run on a web browser, or within mobile applications ("apps") that run on mobile computing devices (e.g., mobile phones, smartphones, tablet computers, wearable devices (watches, armbands, jewelry, hats, goggles, glasses, etc.), laptop computers, etc.). In one example, a client / server architecture may be used, for example, where a mobile computing device (as a client device) sends user input data to a server device and receives final output data (e.g., for display) from the server. In another example, all computations may be performed within a mobile app (and / or other app) on a mobile computing device. In yet another example, computations may be divided between a mobile computing device and one or more server devices.

[0144] Exemplary items Item 1. A computer-implemented method for voice chat translation in a virtual metaverse, comprising the steps of: receiving a request to translate audio relating to the chat functionality of a metaverse place in the virtual metaverse, wherein the audio is received from a first user of a plurality of users, and the plurality of users are associated with a metaverse place; retrieving translation data relating to a second user of a plurality of users, wherein the translation data includes language preferences relating to at least the second user, and the second user is associated with a user device; converting the audio received from the first user into text, wherein the audio includes input speech in a first language spoken by the first user; translating the text into a second language, wherein the second language is defined by language preferences, and the translated text includes context data from the audio; converting the translated text into output speech including the context data; and providing the output speech to a user device.

[0145] Section 2. Subject of any preceding section in which the request identifies a first user and a second user, and the request originates from a computing device associated with the first user.

[0146] Item 3. The subject of any preceding item, further comprising the step of retrieving voice output preferences relating to a first user, wherein the voice output preferences at least partially override translation data relating to a second user.

[0147] Section 4. The subject of any preceding section, which further includes the step of moderating the text to remove words based on a text moderation filter before translating the text into a second language.

[0148] Section 5. Subject matter of any preceding section in which the translation step includes providing text as input to a trained machine learning model and receiving translated text as output from the trained machine learning model.

[0149] Item 6. The subject of any preceding item, whose contextual data includes sentiment data extracted from audio.

[0150] Section 7. The subject of any preceding section, further including the step of preprocessing audio to extract sentiment data.

[0151] Section 8. The subject of any preceding section, wherein the step of converting translated text into output speech includes using a speech waveform modulator to create a modified speech waveform that includes at least partially contextual data.

[0152] Section 9. The subject of any preceding section, further comprising the steps of translating text into multiple different languages ​​to produce multiple different translated texts, and converting the multiple different translated texts into multiple different output speeches.

[0153] Item 10. The subject of any preceding item, comprising the step of providing multiple different output speeches to multiple user devices, each output speech further comprising the step of including utterances in the language associated with each of the multiple user devices.

[0154] Item 11. A non-temporary computer-readable medium storing instructions causing the processing device to perform actions including receiving a request to translate audio related to the chat function of a metaverse place of a virtual metaverse, the audio being received from a first user of a plurality of users and associated with a metaverse place, the plurality of users retrieving translation data related to a second user of a plurality of users, the translation data including language preferences related to at least the second user and associated with a user device, converting the audio received from the first user into text, the audio including input speech in a first language spoken by the first user, translating the text into a second language, the second language defined by language preferences and the translated text including context data from the audio, converting the translated text into output speech including context data, and providing the output speech to the user device.

[0155] Section 12. Subject of any preceding section in which the request identifies a first user and a second user, and the request originates from a computing device associated with the first user.

[0156] Section 13. Subject matter of any preceding section, wherein the operation is to retrieve voice output preferences relating to a first user, further including retrieving voice output preferences that at least partially override translation data relating to a second user.

[0157] Section 14. Subject matter of any preceding section, where the operation further includes moderating the text to remove words based on a text moderation filter before translating the text into a second language.

[0158] Section 15. The subject of any preceding section in which translation includes providing text as input to a trained machine learning model and receiving translated text as output from the trained machine learning model.

[0159] Item 16. The subject of any preceding item, whose contextual data includes sentiment data extracted from audio.

[0160] Section 17. The subject of any preceding section, whose operation further includes preprocessing audio to extract emotional data.

[0161] Section 18. The subject of any preceding section, which involves converting translated text into output speech, including the use of a speech waveform modulator to create a modified speech waveform that includes, at least partially, contextual data.

[0162] Section 19. Subject matter of any preceding section, wherein the operation further includes translating text into multiple different languages ​​to create multiple different translated texts, converting the multiple different translated texts into multiple different output speeches, and providing the multiple different output speeches to multiple user devices, each output speech including utterances in the language associated with each of the multiple user devices.

[0163] Item 20. A system comprising a memory storing instructions and a processing device coupled to the memory and operable to access the memory, wherein when an instruction is executed by the processing device, the processing device receives a request to translate audio related to the chat functionality of a metaverse place of a virtual metaverse, the audio being received from a first user of a plurality of users and associated with a metaverse place, the plurality of users receiving the audio, retrieving translation data related to a second user of a plurality of users, the translation data including language preferences related to at least the second user and associated with a user device, retrieving the audio received from the first user into text, the audio including input speech in a first language spoken by the first user, translating the text into a second language, the second language being defined by language preferences and the translated text including context data from the audio, translating the translated text into output speech including context data, and providing the output speech to the user device.

[0164] conclusion While the explanations were given in relation to specific implementations, these specific implementations are illustrative and not limiting. Concepts shown in the examples may apply to other examples and implementations.

[0165] Where a particular implementation considered herein may acquire or use user data (e.g., user demographics, user behavior data on the platform, user search history, purchased and / or viewed items, user social relationships on the platform, etc.), the user will be provided with choices to control whether, and how, such information is collected, stored, or used. That is, the implementation considered herein will collect, store, and / or use user information only with the explicit authorization of the user and in compliance with applicable regulations.

[0166] Users are enabled to control whether a program or feature collects user information about that particular user or other users associated with that program or feature. Each user from whom information should be collected is presented with options (e.g., through the user interface) that allow the user to exercise control over information collection related to that user, granting permission or authorization regarding whether information is collected and which parts of the information should be collected. Furthermore, certain data may be modified in one or more ways before being stored or used so that personally identifiable information is removed. For example, a user's identity may be modified (e.g., by substitution using pseudonyms, numbers, etc.) so that personally identifiable information cannot be determined. Another example is a user's geographical location being generalized to a larger area (e.g., city, zip code, state, country, etc.).

[0167] It should be noted that the functional blocks, operations, features, methods, devices, and systems described herein may be integrated into or separated into different combinations of systems, devices, and functional blocks, as known to those skilled in the art. Any preferred programming language and programming technique may be used to implement routines in a particular implementation. Different programming techniques, such as procedural or object-oriented programming techniques, may be used. Routines may be executed on a single processing device or on multiple processors. Steps, operations, or calculations may be presented in a specific order, but the order may be modified in different particular implementations. In some implementations, multiple steps or operations shown herein as sequential may be executed simultaneously. [Explanation of symbols]

[0168] 100 Network Environment 102 Online virtual experience platform, server 104 Virtual Experience Engine 105 Games, Virtual Experiences 106 Voice Chat Translation Component 108 Datastores 110 First client device 112 Virtual Experience Applications 114 users 116 Second client device 118 Virtual Experience Applications 120 users 122 Network 200 Network Environment 202 Communication Server 204 Media Server 206 Chat Service 208 Voice Chat Plugin 210 Relay Servers 212 TURN Server 214 TURN Management Component 232 User Audio Streams 234 Translated Audio Streams 300 Voice Translation Pipelines 312 Machine Translation Systems 314 TTS Voice Generation System Training methods for voice-based machine learning models for each of 400 users 402 Input Voice Chat Audio 404 Voice Chat Server 406 Voice Processing Plugin, Voice Processing 408 Voice pre-processing stage 410 How to Moderate and Adjust Voice Chat 414 users 416 users 418 Developer Users 420 How to control player voice chat output 422 Training Data Injection System 424 Machine Learning Model Evaluator Processor 426 Model Data Store 428 datastores 430 Voice Generation Methods 432 Machine Learning Model Generator Processor 434 Training Data Store 436 Datastores 438 Training Data Cleanup Processor 442 Voice adjustment preprocessor, preprocessing 444 STT System 446 Voice Adjustment 448 Text Moderation and / or Filtering 452 Voice Output Preferences 454 Voice Preference Service 456 Voice Output Generation System 458 Voice Adjustment System 462 datastores 464 datastores 466 datastores 470 Native language information 472 NPC Text Information 474 Voice Characteristic Definition 482 Translated Text 486 Multiple Output Speeches 490 Audio Data Store 500 Voice Machine Learning Model Training Methods 502 User Audio 504 Model Management System 506 Model training methods for each user 508 STT components 510 Text 512 Translated User Audio 516 Translated speech texts 600 ways 700 methods 800 Computing Devices 802 Processor 804 memory 806 I / O Interface 808 Operating System 810 Applications 812 Related data 814 Audio / Video Input / Output Devices

Claims

1. A computer-based method for translating voice chat in a virtual metaverse, A step of receiving a request to translate audio related to the chat function of a metaverse place of the virtual metaverse, wherein the audio is received from a first user among a plurality of users, the plurality of users are associated with the metaverse place, and the audio includes input speech in a first language spoken by the first user. A step of retrieving translation data related to a second user among the plurality of users, wherein the translation data includes at least language preferences related to the second user, and the second user is associated with a user device. The steps of converting audio received from the first user into phonemes of the first language, A step of extracting context data from the audio using the phonemes of the first language, A step of processing the phonemes of the first language in order to predict a match between each phoneme of the first language and each phoneme of the second language, wherein the second language is defined by the language preference, The steps include converting the phonemes of the second language into output speech including the context data, A method performed by a computer, comprising the step of providing the output speech to the user device.

2. A method performed by a computer according to claim 1, wherein the request identifies the first user and the second user, and the request originates from a computing device associated with the first user.

3. A method performed by a computer according to claim 1, further comprising the steps of retrieving a voice output preference relating to the first user, wherein the voice output preference at least partially overrides the translation data relating to the second user.

4. The computer-implemented method according to claim 1, wherein the step of converting the audio to the phonemes of the first language includes the step of converting the audio to text, further comprising the step of moderating the text to remove words based on a text moderation filter before processing the phonemes of the first language to predict the match.

5. The computer-based method according to claim 1, wherein the processing step includes providing the phonemes of the first language as input to a trained machine learning model and receiving the phonemes of the second language as output from the trained machine learning model.

6. The computer-based method according to claim 1, wherein the context data includes emotion data extracted from the audio.

7. The method performed by a computer according to claim 6, further comprising the step of preprocessing the audio to extract the emotion data.

8. The method performed by a computer according to claim 1, wherein the step of converting the phonemes of the second language into output speech includes using a speech waveform modulator to create a modified speech waveform that includes at least the context data.

9. A computer-based method according to claim 1, further comprising the steps of: translating the phonemes of a first language into a plurality of different languages ​​in order to create phonemes of a plurality of different languages; and converting the phonemes of the plurality of different languages ​​into a plurality of different output speeches.

10. A method performed by a computer according to claim 9, further comprising the step of providing a plurality of different output speeches to a plurality of user devices, wherein each output speech includes an utterance in the language associated with each of the plurality of user devices.

11. In response to execution by the processing device, the processing device is instructed to: Receiving a request to translate audio related to the chat function of a metaverse place in a virtual metaverse, wherein the audio is received from a first user among a plurality of users, and the plurality of users are associated with the metaverse place, and the audio includes input speech in a first language spoken by the first user. Retrieving translation data related to a second user among the aforementioned plurality of users, wherein the translation data includes at least language preferences related to the second user, and the second user is associated with a user device, Converting the audio received from the first user into phonemes of the first language, Using the phonemes of the first language, context data is extracted from the audio, Processing the phonemes of the first language in order to predict a match between each phoneme of the first language and each phoneme of the second language, wherein the second language is defined by the language preference, Converting the phonemes of the second language into output speech including the context data, A non-temporary computer-readable medium storing instructions for causing the user device to perform an operation including providing the output speech.

12. The non-temporary computer-readable medium according to claim 11, wherein the request identifies the first user and the second user, and the request originates from a computing device associated with the first user.

13. The non-temporary computer-readable medium according to claim 11, wherein the operation further includes retrieving a voice output preference relating to the first user, wherein the voice output preference at least partially overrides the translation data relating to the second user.

14. The non-temporary computer-readable medium according to claim 11, wherein converting the audio to the phonemes of the first language comprises converting the audio to text, and the operation further comprises moderating the text to remove words based on a text moderation filter before processing the phonemes of the first language to predict the match.

15. The non-temporary computer-readable medium according to claim 11, wherein the processing includes providing the phonemes of the first language as input to a trained machine learning model and receiving the phonemes of the second language as output from the trained machine learning model.

16. The non-temporary computer-readable medium according to claim 11, wherein the context data includes emotion data extracted from the audio.

17. The non-temporary computer-readable medium according to claim 16, wherein the operation further comprises preprocessing the audio to extract the emotion data.

18. The non-temporary computer-readable medium according to claim 11, wherein converting the phonemes of the second language into output speech includes using a speech waveform modulator to create a modified speech waveform that includes at least the context data.

19. The aforementioned operation, Translating the phonemes of the first language into multiple different languages ​​in order to create phonemes of multiple different languages, Converting the phonemes of the aforementioned multiple different languages ​​into multiple different output speeches, The non-temporary computer-readable medium according to claim 11, further comprising providing the plurality of different output speeches to a plurality of user devices, wherein each output speech includes utterances in the language associated with each of the plurality of user devices.

20. It is a system, The memory that stores the instructions, The processing device is coupled to the memory and is operable to access the memory, and when the instruction is executed by the processing device, the processing device is configured to: Receiving a request to translate audio related to the chat function of a metaverse place in a virtual metaverse, wherein the audio is received from a first user among a plurality of users, and the plurality of users are associated with the metaverse place, and the audio includes input speech in a first language spoken by the first user. Retrieving translation data related to a second user among the aforementioned plurality of users, wherein the translation data includes at least language preferences related to the second user, and the second user is associated with a user device, Converting the audio received from the first user into phonemes of the first language, Using the phonemes of the first language, context data is extracted from the audio, Processing the phonemes of the first language in order to predict a match between each phoneme of the first language and each phoneme of the second language, wherein the second language is defined by the language preference, Converting the phonemes of the second language into output speech including the context data, To provide the output speech to the user device, A system that performs actions including those mentioned above.

Images