Cross-modal training of machine learning models to identify misuse of audio streams

Abstract

The metaverse application receives a user-provided audio stream related to the user. The metaverse application retrieves one or more parts of the audio stream. The metaverse application divides the user-provided audio stream into multiple parts, each part corresponding to a specific time window of the audio stream. The metaverse application provides these multiple parts of the user-provided audio stream as input to an audio machine learning model. Based on the parts of the user-provided audio stream, the audio machine learning model outputs a decision on whether a particular part of the multiple parts has been abused. The metaverse application takes corrective action in response to the decision on the abuse of a particular part.

Description

[[TECHNICAL FIELD]]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a non-provisional application claiming priority under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No. 63 / 534,086, filed August 22, 2023, U.S. Provisional Patent Application No. 63 / 614,697, filed December 26, 2023, and U.S. Provisional Patent Application No. 63 / 563,240, filed March 8, 2024, the entire contents of each of which are incorporated herein by reference.

[0002] Embodiments generally relate to online virtual experience platforms, and more particularly, to methods, systems, and computer-readable media for identifying abuse of audio streams. [[BACKGROUND ART]]

[0003] Abuse of a virtual environment occurs in various ways. For example, avatars may wear offensive clothing, players may engage in offensive behavior, players may say offensive (abusive) things in audio chat, and / or players may type offensive words in group chat. Moderating audio in real-time communication is difficult when numerous players are participating in the virtual environment due to the amount of audio that moderators must review in a short period of time. The longer the delay between a violation and disciplinary action in response to the violation, the higher the likelihood that the player will continue to engage in abuse of the virtual environment.

[0004] The background description provided herein is for the purpose of presenting the context of the present disclosure. To the extent described in this background section, the work of the inventors currently mentioned, and in some cases aspects of the description that do not qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art to the present disclosure. [[SUMMARY OF THE INVENTION]] [Means for solving the problem]

[0005] A computer implementation method performs moderation of an audio stream. This method includes the step of receiving a user-provided audio stream related to a user. This method further includes the step of dividing the user-provided audio stream into multiple parts, each part corresponding to a specific time window of the audio stream. This method further includes the step of providing the multiple parts of the user-provided audio stream as input to an audio machine learning model. This method further includes the step of having the audio machine learning model output a decision on the abuse of a particular part of the multiple parts based on the parts of the user-provided audio stream. This method further includes the step of taking corrective action in response to the decision on the abuse of a particular part.

[0006] In some embodiments, an audio machine learning model is trained by: providing an audio input to an audio encoder; having the audio encoder output an audio hazard classification that identifies an audio embedding corresponding to the audio input and one or more hazardous labels to associate with the audio input; providing a text input to a text encoder, wherein the text input is a transcription of the audio input; having the text encoder output a text embedding based on the text input; determining a value for a text injection loss function based on a comparison of the audio embedding and the text embedding; and adjusting one or more parameters of the audio encoder to reduce the value of the text injection loss function.

[0007] In some embodiments, the audio input includes real-world audio related to abuse reports from one or more users, and the method further includes the steps of: comparing an audio toxicity classification to a label associated with the real-world audio in order to determine a value of a classifier loss function, wherein the label associated with the real-world audio is ground truth provided by a human judge; and adjusting the parameters of an audio encoder to reduce the value of the classifier loss function. In some embodiments, an audio machine learning model is trained using training data, and the method further includes the steps of: receiving a training audio stream of one or more people speaking; for each training audio stream, dividing the training audio stream into two or more audio segments; transcribing the two or more audio segments into two or more text segments; generating a first segment label for each of the two or more text segments using a first classifier, wherein the first segment label indicates whether the text segment is toxic or harmless; and adding the training audio stream, the two or more text segments, and the corresponding first segment labels from the training audio stream to a training dataset. In some embodiments, the step of generating training data further includes identifying a subset of the training audio stream from the training audio stream, wherein one or more of a first segment labels indicate that one or more of the text segments are malicious; generating a second segment label for the subset of the training audio stream using a second classifier, wherein the second classifier identifies instances of abuse more accurately than the first classifier; and adding the second segment label to the training set.

[0008] In some embodiments, an audio machine learning model is trained using synthetic training data, and this method further includes generating synthetic training data by: providing voice chat audio to an automatic speech recognition (ASR) system; having the ASR system output transcribed audio based on the voice chat audio; providing the transcribed audio and prompts identifying new text characteristics to a large language model (LLM), configured to generate new text based on the prompts and transcribed audio; providing the voice chat audio to a voice kroner that outputs audio tokens storing speaker characteristics within the voice chat audio; providing the new text and audio tokens as input to a text-to-speech system; and having the text-to-speech system output synthetic training.

[0009] In some embodiments, the corrective action includes giving a warning to the user. In some embodiments, the corrective action includes muting the microphone on a user device associated with the user, or preventing the user-provided audio stream from being distributed to one or more other users. In some embodiments, the abuse determination includes identifying the type of abuse, which is selected from one or more groups of blasphemy, bullying, harassment, and sexism, as well as combinations thereof.

[0010] A system for moderating an audio stream, the system comprising one or more processors and one or more processor-bound memories storing instructions thereon, the instructions causing the processors to perform actions when executed by the processors. These actions include receiving a user-provided audio stream related to a user; dividing the user-provided audio stream into multiple parts, each part corresponding to a specific time window of the audio stream; providing the multiple parts of the user-provided audio stream as input to an audio machine learning model; the audio machine learning model outputting a decision on the abuse of a particular part of the multiple parts based on the parts of the user-provided audio stream; and taking corrective action in response to the decision on the abuse of a particular part.

[0011] In some embodiments, an audio machine learning model is trained by providing an audio input to an audio encoder, by having the audio encoder output an audio toxicity classification that identifies an audio embedding corresponding to the audio input and one or more toxicity labels to associate with the audio input based on the audio input, by providing a text input to a text encoder, wherein the text input is a transcription of the audio input, by having the text encoder output a text embedding based on the text input, by determining a value for a text injection loss function based on a comparison of the audio embedding and the text embedding, and by adjusting one or more parameters of the audio encoder to reduce the value of the text injection loss function.

[0012] In some embodiments, the audio input includes real-world audio related to abuse reports from one or more users, and these operations further include comparing the audio toxicity classification to labels associated with the real-world audio in order to determine the value of the classifier loss function, wherein the labels associated with the real-world audio are ground truth provided by human judges, and adjusting the parameters of the audio encoder to reduce the value of the classifier loss function. In some embodiments, the audio machine learning model is trained using training data, and these operations further include receiving training audio streams of one or more people speaking, for each training audio stream, splitting the training audio stream into two or more audio segments, transcribing the two or more audio segments into two or more text segments, generating a first segment label for each of the two or more text segments using a first classifier, wherein the first segment label indicates whether the text segment is harmful or harmless, and generating training data by adding the training audio streams, the two or more text segments, and the corresponding first segment labels from the training audio streams to a training dataset.

[0013] In some embodiments, an audio machine learning model is trained using training data, and this method further includes the steps of: receiving a training audio stream of one or more people speaking; for each training audio stream, dividing the training audio stream into two or more audio segments; transcribing the two or more audio segments into two or more text segments; generating a first segment label for each of the two or more text segments using a first classifier, wherein the first segment label indicates whether the text segment is harmful or harmless; and generating training data by adding the training audio stream, the two or more text segments, and the corresponding first segment labels from the training audio stream to a training dataset. In some embodiments, the step of generating training data further includes identifying a subset of the training audio stream from the training audio stream, wherein one or more of a first segment labels indicate that one or more of the text segments are malicious; generating a second segment label for the subset of the training audio stream using a second classifier, wherein the second classifier identifies instances of abuse more accurately than the first classifier; and adding the second segment label to the training set.

[0014] A non-temporary computer-readable medium that stores instructions, which, when executed by one or more computers, stores instructions on which one or more computers perform actions. These actions include receiving a user-provided audio stream related to a user; dividing the user-provided audio stream into multiple parts, each part corresponding to a specific time window of the audio stream; providing the multiple parts of the user-provided audio stream as input to an audio machine learning model; having the audio machine learning model output a decision on the abuse of a particular part of the multiple parts based on the parts of the user-provided audio stream; and taking corrective action in response to the decision on the abuse of a particular part.

[0015] In some embodiments, an audio machine learning model is trained by providing an audio input to an audio encoder, by having the audio encoder output an audio toxicity classification that identifies an audio embedding corresponding to the audio input and one or more toxicity labels to associate with the audio input based on the audio input, by providing a text input to a text encoder, wherein the text input is a transcription of the audio input, by having the text encoder output a text embedding based on the text input, by determining a value for a text injection loss function based on a comparison of the audio embedding and the text embedding, and by adjusting one or more parameters of the audio encoder to reduce the value of the text injection loss function.

[0016] In some embodiments, the audio input includes real-world audio related to abuse reports from one or more users, and these operations further include comparing the audio toxicity classification to labels associated with the real-world audio in order to determine the value of the classifier loss function, wherein the labels associated with the real-world audio are ground truth provided by human judges, and adjusting the parameters of the audio encoder to reduce the value of the classifier loss function. In some embodiments, the audio machine learning model is trained using training data, and these operations further include receiving a training audio stream of one or more people speaking, for each training audio stream, splitting the training audio stream into two or more audio segments, transcribing the two or more audio segments into two or more text segments, generating a first segment label for each of the two or more text segments using a first classifier, wherein the first segment label indicates whether the text segment is harmful or harmless, and generating training data by adding the training audio stream, the two or more text segments, and the corresponding first segment labels from the training audio stream to a training dataset. In some embodiments, the corrective action includes giving a warning to the user. [Brief explanation of the drawing]

[0017] [Figure 1] This is a block diagram of an exemplary network environment according to some embodiments described herein. [Figure 2] This is a block diagram of an exemplary computing device according to some embodiments described herein. [Figure 3A] This is an illustrative diagram of the process for creating labels for training data according to some embodiments described herein. [Figure 3B]1 is an exemplary diagram of a process for generating labels for training data according to some embodiments described herein. [Figure 3C] 3 is an exemplary diagram of how portions of audio are segmented by a machine learning model according to some embodiments described herein. [Figure 4] 6 is an example block diagram of a keyword classifier, a background noise classifier, and a segment-level classifier according to some embodiments described herein. [Figure 5] 9 is a diagram illustrating an exemplary architecture of an audio machine learning model that generates synthetic audio according to some embodiments described herein. [Figure 6A] 12 is a diagram illustrating an exemplary architecture of an audio machine learning model that identifies harmful content in an audio stream according to some embodiments described herein. [Figure 6B] 15 is a diagram illustrating an exemplary architecture of an audio machine learning model that identifies harmful content in an audio stream according to some embodiments described herein. [Figure 6C] 18 is a diagram illustrating an exemplary architecture of an audio machine learning model that identifies harmful content in an audio stream according to some embodiments described herein. [Figure 6D] 21 is a diagram illustrating an exemplary architecture of an audio machine learning model that identifies harmful content in an audio stream according to some embodiments described herein. [Figure 7] 24 is a diagram illustrating an exemplary architecture of an audio machine learning system that identifies harmful content in an audio stream according to some embodiments described herein. [Figure 8] 27 is a diagram illustrating an exemplary process for moderating an audio stream according to some embodiments described herein. [Figure 9]FIG. 1 illustrates an example user interface that warns a user that the user's utterance may result in a corrective action, in accordance with some embodiments described herein. [Figure 10] FIG. 2 is a flow diagram of an example method for generating training data for an audio machine learning model, in accordance with some embodiments described herein. [Figure 11] FIG. 3 is a flow diagram of an example method for training an audio machine learning model to identify harm in an audio stream, in accordance with some embodiments described herein. [Figure 12] FIG. 4 is a flow diagram of an example method for moderating an audio stream, in accordance with some embodiments described herein. DETAILED DESCRIPTION OF EMBODIMENTS

[0018] When users interact on a virtual environment platform, a first user may engage in several types of abuse. Abuse of voice chat has previously been difficult to detect in real time due to the inherent latency that exists when waiting for a moderator to review an audio stream, for example after a user report of abuse, and make a determination as to whether the audio stream contained abuse.

[0019] Real-time audio machine learning models have been difficult to implement because if an audio machine learning model outputs too many false positives (i.e., identifies too many instances of abuse that are not actually abuse), it can annoy users and discourage them from interacting in the virtual experience. If an audio machine learning model outputs too many false negatives (i.e., fails to identify instances of abuse), it can make the virtual experience unsafe because too many users may be exposed for abusive purposes. In addition, audio machine learning models can suffer from a lack of diversity in training data if certain types of labeled training data (e.g., audio associated with racist labels) are insufficient to create a robust audio machine learning model with sufficient accuracy / recall.

[0020] This disclosure, in particular, describes a metaverse application that uses an audio machine learning model to identify abuse of audio streams in a virtual environment in real time. The audio machine learning model has high accuracy (accurately classifying audio as malicious and non-malicious) along with good recall (detecting the majority of malicious audio). The audio machine learning model is small in size (e.g., memory requirements) and computationally efficient.

[0021] In some embodiments, an audio machine learning model is trained by using a first segment classifier that identifies in real time whether an audio stream is harmful or harmless based on the semantic properties of the segments. Training may also include using a keyword classifier that identifies whether each word is harmful or harmless, and integrating the results from the first segment classifier and the keyword classifier for improved recognition of harmfulness. Training may also include a second segment classifier that takes longer to process the audio stream but identifies harmfulness within the audio stream more accurately.

[0022] In some embodiments, audio machine learning models are trained using synthetic data containing labels paired with harmful labels. In some embodiments, audio machine learning models are trained using synthetic audio, which is generated by providing transcribed audio to a large language model from a voice chat, along with prompts requesting new text with specific characteristics. The new text is combined with the original speaker characteristics to create the synthetic audio. Synthetic audio may be used to provide greater diversity to the training data by serving as a source of training data for categories of audio with smaller datasets and / or for categories of audio where labeling is more difficult because a larger amount of data is required to determine the context of the audio.

[0023] Once the audio machine learning model is trained, it is used for real-time content moderation. The method may include the step of receiving a user-provided audio stream related to a user. For example, the user may be a player in a virtual environment. The user-provided audio stream is divided into multiple parts, each part corresponding to a specific time window of the audio stream, such as every 15 seconds or every time the user pauses. Multiple parts of the user-provided audio stream are provided as input to the audio machine learning model, which outputs a decision on the abuse. Corrective actions are taken in response to the decision on the abuse of a particular part of the user-provided audio stream. For example, a first attack may include a warning, a subsequent attack may include muting the user's microphone, and a more serious attack may include prohibiting the user from participating in the virtual environment for a certain period of time.

[0024] Exemplary network environment Figure 1 shows an exemplary network environment 100 in several implementations of the present disclosure. Figure 1 and other figures use similar reference numbers to identify similar elements. Letters following a reference number, such as "110a," indicate that the text specifically refers to the element that has that particular reference number. A reference number in text that is not followed by letters, such as "110," refers to one or all of the elements in the figure that have that reference number (for example, "110" in text refers to reference numbers "110a," "110b," and / or "110n" in the figure).

[0025] The network environment 100 (also referred to herein as the “Platform”) includes an online virtual experience server 102, a data store 108, and a client device 110 (or more client devices), all connected to network 122.

[0026] The online virtual experience server 102 may include, among other things, a virtual experience engine 104, one or more virtual experiences 105, and a moderation application 130. In some implementations, the online virtual experience server 102 may be configured to provide the virtual experience 105 to one or more client devices 110 and to moderate the audio stream via the moderation application 130.

[0027] Although the data store 108 is shown coupled to the online virtual experience server 102, in some implementations it may also be provided as part of the online virtual experience server 102. In some implementations, the data store may be configured to store advertising data, user data, engagement data, and / or other contextual data related to the moderation application 130.

[0028] Client devices 110 (e.g., 110a, 110b, 110n) may interact with the online virtual experience server 102 and include virtual experience applications 112 (e.g., 112a, 112b, 112n) and I / O interfaces 114 (e.g., 114a, 114b, 114n) for viewing, for example, a graphical user interface (GUI) through a computer monitor or display (not shown). In some implementations, client devices 110 may be configured to conduct and display a virtual experience, which may include virtual user engagement portals as described herein.

[0029] The network environment 100 is provided for illustrative purposes. In some implementations, the network environment 100 may include the same, fewer, more, or different elements, configured in the same or different ways as shown in Figure 1.

[0030] In some implementations, network 122 may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wired network (e.g., an Ethernet network), a wireless network (e.g., an 802.11 network, a Wi-Fi® network, or a wireless LAN (WLAN)), a cellular network (e.g., a Long Term Evolution (LTE) network), routers, hubs, switches, server computers, or a combination thereof.

[0031] In some implementations, the datastore 108 may be non-temporary computer-readable memory (e.g., random access memory), a cache, a drive (e.g., a hard drive), a flash drive, a database system, or another type of component or device capable of storing data. The datastore 108 may include multiple storage components (e.g., multiple drives or multiple databases) that may span multiple computing devices (e.g., multiple server computers).

[0032] In some implementations, the online virtual experience server 102 may include a server having one or more computing devices (e.g., a cloud computing system, a rack-mount server, a server computer, a cluster of physical servers, a virtual server, etc.). In some implementations, the server may be contained within the online virtual experience server 102, may be a separate system, or may be part of another system or platform. In some implementations, the online virtual experience server 102 may be a single server, or any combination of multiple servers, load balancers, network devices, and other components. The online virtual experience server 102 may be implemented on a physical server, but in some implementations, virtualization technology may be utilized. Other variants of the online virtual experience server 102 are also applicable.

[0033] In some implementations, the online virtual experience server 102 may include one or more computing devices (such as rack-mount servers, router computers, server computers, personal computers, mainframe computers, laptop computers, tablet computers, and desktop computers), data stores (e.g., hard disks, memory, databases), networks, software components, and / or hardware components that operate on the online virtual experience server 102 and are used to provide users (e.g., users 114 via client devices 110) with access to the online virtual experience server 102.

[0034] The online virtual experience server 102 may include application backend software that may be used to provide users with access to a website (for example, one or more web pages) or content provided by the online virtual experience server 102. For example, users (or developers) may access the online virtual experience server 102 using a virtual experience application 112 on a client device 110.

[0035] In some implementations, the online virtual experience server 102 may include digital assets and digital virtual experience generation provisions. For example, the platform may provide an administrator interface that enables design, modification, personalized customization, and other modification functions. In some implementations, the virtual experience may include, for example, a two-dimensional game, a three-dimensional game, a virtual reality (VR) game, or an augmented reality (AR) game. In some implementations, the virtual experience creator and / or developer may search for virtual experiences, combine parts of virtual experiences, tailor virtual experiences for specific activities (e.g., group virtual experiences), and other features provided through the virtual experience server 102.

[0036] In some implementations, the online virtual experience server 102 or client device 110 may include a virtual experience engine 104 or a virtual experience application 112. In some implementations, the virtual experience engine 104 may be used for developing or running a virtual experience 105. For example, the virtual experience engine 104 may include, among other things, 2D graphics, 3D graphics, VR graphics, or AR graphics, a physics engine, a collision detection engine (and collision response), a sound engine, scripting capabilities, a haptic engine, an artificial intelligence engine, networking capabilities, streaming capabilities, memory management capabilities, threading capabilities, scene graph capabilities, or a rendering engine ("renderer") for video support for movies. Components of the virtual experience engine 104 may generate commands (e.g., rendering commands, collision commands, physics commands, etc.) that help compute and render the virtual experience.

[0037] The online virtual experience server 102, which uses the virtual experience engine 104, may perform some or all of the virtual experience engine functions (e.g., generate physical commands, rendering commands, etc.) or offload some or all of the virtual experience engine functions to the virtual experience engine 104 of the client device 110 (not shown). In some implementations, each virtual experience 105 may have a different ratio between the virtual experience engine functions performed on the online virtual experience server 102 and the virtual experience engine functions performed on the client device 110.

[0038] In some implementations, virtual experience instructions may refer to instructions that enable the client device 110 to render the gameplay, graphics, and other features of the virtual experience. These instructions may include one or more of the following: user input (e.g., physical object placement), character placement and velocity information, or commands (e.g., physics commands, rendering commands, collision commands, etc.).

[0039] In some implementations, the client devices 110 may each include computing devices such as personal computers (PCs), mobile devices (e.g., laptops, mobile phones, smartphones, tablet computers, or netbooks), network-connected televisions, and game consoles. In some implementations, the client devices 110 may also be referred to as "client device 110". In some implementations, one or more client devices 110 may connect to the online virtual experience server 102 at a given time. Note that the number of client devices 110 is provided as an example and is not limited. In some implementations, any number of client devices 110 may be used.

[0040] In some implementations, each client device 110 may contain an instance of the virtual experience application 112. The virtual experience application 112 may be rendered for interaction on the client device 110. While user interaction is taking place in the virtual experience of the online platform 100 or another GUI, the user may create avatars containing different body parts from different libraries. The moderation application 130 may use an audio stream from a user participating in the virtual experience as input and identify instances of harmful behavior in the audio stream. The moderation application 103 may warn the user about the harmful behavior and take corrective action in response to the identification of instances of harmful behavior, such as muting the user's audio.

[0041] Exemplary computing device Figure 2 is a block diagram of an exemplary computing device 200 that may be used to implement one or more features described herein. The computing device 200 may be any suitable computer system, server, or other electronic or hardware device. In some embodiments, the computing device 200 is a client device 110. In some embodiments, the computing device 200 is an online virtual experience server 102.

[0042] In some embodiments, the computing device 200 includes a processor 235, memory 237, input / output (I / O) interface 239, microphone 241, speaker 243, display 245, and storage device 247, all coupled via a bus 218. In some embodiments, the computing device 200 includes additional components not shown in Figure 2. In some embodiments, the computing device 200 includes fewer components than those shown in Figure 2. For example, if the moderation application 130 is stored on the online virtual experience server 102 in Figure 1, the computing device may not include the microphone 241, speaker 243, or display 245.

[0043] The processor 235 may be connected to the bus 218 via signal line 222, the memory 237 may be connected to the bus 218 via signal line 224, the I / O interface 239 may be connected to the bus 218 via signal line 226, the microphone 241 may be connected to the bus 218 via signal line 228, the speaker 243 may be connected to the bus 218 via signal line 230, the display 245 may be connected to the bus 218 via signal line 232, and the storage device 247 may be connected to the bus 218 via signal line 234.

[0044] The processor 235 includes an arithmetic logic unit, a microprocessor, a general-purpose controller, or several other processor arrays for performing calculations and providing instructions to the display device. The processor 235 may include various computing architectures, including complex instruction set computer (CISC) architectures, reduced instruction set computer (RISC) architectures, or architectures that process data and implement combinations of instruction sets. In some implementations, the processor 235 may include dedicated units, such as a machine learning processor, an audio / video coding and decoding processor, etc. Figure 2 shows a single processor 235, but multiple processors 235 may be included. In different embodiments, the processor 235 may be a single-core processor or a multi-core processor. Other processors (e.g., a graphics processing unit), an operating system, sensors, a display, and / or physical configurations may be parts of the computing device 200, such as a keyboard and mouse.

[0045] Memory 237 stores instructions that may be executed by processor 235 and / or data. These instructions may include code and / or routines for performing the techniques described herein. Memory 237 may be a dynamic random access memory (DRAM) device, static RAM, or some other memory device. In some embodiments, memory 237 also includes non-volatile memory such as static random access memory (SRAM) devices or flash memory, or similar persistent storage devices and media including hard disk drives, compact disc read-only memory (CD-ROM) devices, DVD-ROM devices, DVD-RAM devices, DVD-RW devices, flash memory devices, or some other mass storage devices for more persistent storage of information. Memory 237 includes code and routines that can operate to perform the moderation application 130, which is described in more detail below.

[0046] The I / O interface 239 may provide functionality to enable the computing device 200 to interface with other systems and devices. The interfaced devices may be included as part of the computing device 200 or communicate with the computing device 200 separately. For example, network communication devices, storage devices (e.g., memory 237 and / or storage device 247), and input / output devices may communicate via the I / O interface 239. In another example, the I / O interface 239 may receive data from the online virtual experience server 102 and distribute the data to the moderation application 130 and its components, such as the user interface module 202. In some embodiments, the I / O interface 239 may connect to input devices (such as a keyboard, pointing device, touchscreen, microphone 241, and sensors) and / or output devices (such as a display 245 and speaker 243), as well as to the interface device.

[0047] Some examples of interfaced devices that may be connected to the I / O interface 239 may include a display 245 that can be used to display content, such as images, videos, and / or a metaverse user interface as described herein, and to receive touch (or gesture) input from the user. The display 245 may include any suitable display device, such as a liquid crystal display (LCD), light-emitting diode (LED), plasma display screen, cathode ray tube (CRT), television, monitor, touchscreen, 3D display screen, projector (e.g., 3D projector), or other visual display device.

[0048] Microphone 241 includes hardware, such as one or more microphones that detect audio spoken by a person. Microphone 241 may transmit audio to the moderation application 130 via the I / O interface 239.

[0049] Speaker 243 includes hardware for generating audio for playback. In some embodiments, speaker 243 may also include audio hardware that supports playback via an external, separate speaker coupled to the computing device 200 (e.g., wired or wireless headphones, external speakers, or other audio playback devices).

[0050] The storage device 247 stores data related to the moderation application 130. For example, the storage device 247 may store user profiles associated with user 125, a list of blocked avatars, and so on.

[0051] Exemplary Moderation Application Figure 2 shows a computing device 200 running an exemplary moderation application 130, which includes a user interface module 202, a speech recognition engine 204, an audio machine learning module 206, and an abuse detection module 208. In some embodiments, a single computing device 200 includes all the components shown in Figure 2. In some embodiments, one or more of the components may reside on different computing devices 200. For example, a client device 110 may include the user interface module 202, and the abuse detection module 208 may be implemented on the online virtual experience server 102. In some embodiments, one or more different parts of modules 202, 204, 206, and 208 may be implemented on the client device 110 or on the online virtual experience server 102. In some embodiments, modules 202, 204, 206, and 208 may be used during the training of a machine learning model for abuse detection, and modules 202 and 208 are used during the interface (i.e., while the moderation application 130 performs abuse detection).

[0052] The user interface module 202 generates graphical data for displaying a user interface for a user associated with the client device 110 to participate in a virtual experience. In some embodiments, before the user participates in the virtual experience, the user interface module 202 generates a user interface that includes information about how the user's information may be collected, stored, and / or analyzed. For example, the user interface may require the user to provide permission to use any information related to the user. The user may be informed that their information may be deleted by the user, and the user may have the option to choose what types of information are provided for different uses. The use of the information is subject to applicable provisions, and the data is stored securely. Data collection is not performed in a fixed location or against a fixed user category (e.g., based on age or other demographics), data collection is temporary (i.e., the data is discarded after a certain period of time), and the data is not shared with third parties. Some of the data may be anonymized, aggregated across users, or, in some cases, modified so that specific user identification information cannot be determined.

[0053] The user interface module 202 receives user input from the user during gameplay of the virtual experience. For example, based on user input, the avatar may move around, take actions, change its posture, and (via audio chat) talk to other users in the virtual experience. The user interface module 202 generates graphical data to display the avatar's location, actions, posture, etc., in the virtual experience.

[0054] Users may interact with other users in the virtual experience. Some of these interactions may be negative, and in some embodiments, the user interface module 202 generates graphical data for the user interface that allows the user to limit their exposure to other users they wish to avoid. For example, the user interface module 202 may include options to mute other users, block other users, and report abuse occurring in the virtual experience. For example, another avatar may be wearing inappropriate clothing, an avatar may be holding an inappropriate object (e.g., a flag associated with a hate group, an object of some offensive shape, etc.), an avatar may be performing offensive behavior (e.g., an avatar may be using spray paint to paint an image of genitals), or an avatar may be making inappropriate remarks (e.g., either in a chat box or directly via voice chat to the user). One avatar may be associated with multiple types of abuse, such as performing offensive behavior while simultaneously wearing inappropriate clothing.

[0055] As described in detail below, the abuse detection module 208 determines that a user has committed an abuse and, in response to this determination, takes corrective action. In some embodiments, the corrective action includes giving the user a warning. The abuse detection module 208 may instruct the user interface module 202 to generate graphical data for displaying the warning. In some embodiments, the corrective action includes muting the microphone on the client device associated with the user. In some embodiments, instead of muting the microphone on the client device, the abuse detection module 208 suppresses the distribution of the user-provided audio stream to one or more other users. The abuse detection module 208 instructs the user interface module 202 to generate graphical data to explain to the user that their microphones are muted, how long the microphones have been muted, and how further violations would have more extreme consequences.

[0056] The speech recognition engine 204 receives an audio stream. The audio stream used to generate training data for the audio machine learning model is used subject to user consent. User consent is lawfully obtained from the user and authorized to grant permission according to the regulations (e.g., a parent for users under a certain age, or the user themselves for users of a certain age). User-identifiable information is removed from the audio stream. Audio streams collected using secure communications are not stored once the training of the audio machine learning model is complete. In some embodiments, only instances of user-provided audio, including examples of misuse, are used for training. All applicable laws and regulations regarding user-provided audio are complied with.

[0057] In some embodiments, the speech recognition engine 204 processes the audio stream by applying different filters to the audio stream. In some embodiments, the speech recognition engine 204 includes a voice activity detection (VAD) model that verifies that human speech is detected within the audio stream. The VAD model advantageously reduces instances where content moderation is performed on audio streams that do not contain human speech. As a result, the content moderation process is computationally more efficient and faster (by eliminating non-human speech portions of the audio). For example, queries for inference to determine whether a particular portion of audio contains malicious content can be significantly reduced by excluding non-human speech audio, and query volume may be reduced by, for example, 10% in some cases.

[0058] In some embodiments, the speech recognition engine 204 applies a filter to remove background noise from the audio stream. As a result, detecting human speech within the audio stream becomes even easier.

[0059] The speech recognition engine 204 (for example, an automatic speech recognition (ASR) engine) divides the audio stream into parts. For example, these parts may be 15 seconds or less, 30 seconds or less, up to 60 seconds, or of other lengths. The speech recognition engine 204 divides these parts into segments. In some embodiments, the speech recognition engine 204 divides segments based on the detection of pauses between words, which may correlate with divisions between parts of a sentence or between sentences. The speech recognition engine 204 generates a transcript for each part.

[0060] The audio machine learning module 206 trains an audio machine learning model to output abuse decisions. In some embodiments, the audio machine learning model is trained to detect and classify one or more of the following: offensive keywords, offensive segments, and disruptive noise and emotion.

[0061] The audio machine learning module 206 generates a training dataset. In some embodiments, the training dataset includes synthetic audio data. The use of synthetic audio data advantageously solves the problems associated with easily confused words (e.g., ship / shit, ask / ass, fudge / fuck, flag / fag, etc.). In some embodiments, the synthetic data includes human voice data with synthetic labels, and synthetic speech from text chats where the text is identified as containing instances of malice. Human voice data is used only when the user provides consent for the use of that data to train the audio machine learning model. If the user does not provide consent for the use of that data, there is no impact on platform participation.

[0062] In some embodiments, the training dataset also includes ground truth labeled data, such as when a moderator reviews the raw voice data and assigns labels to different parts, for example, as malicious (optionally, with an abuse category) or non-malicious. In some embodiments, ground truth labels may also be derived from abuse reports, such as when a user expresses dissatisfaction with another user's behavior. Limited personnel may be used to apply labels to audio submitted with abuse reports in order for the audio machine learning model to generate synthetic training audio data and corresponding labels, and these abuse reports may then be used as ground truth data to compare with the labels generated by the audio machine learning model and used to train the audio machine learning model.

[0063] Referring to Figure 3A, an example 300 of the process for creating labels for training data according to some embodiments described herein is shown. A portion of the audio stream is received (305). This portion may be defined temporarily (for example, every 15 seconds) based on interruptions to speech, etc.

[0064] In this example, the portion of the audio stream is "Hey, you. That *woman* is *f*cked" without the asterisk. The speech recognition engine 204 divides that portion of the audio stream into pairs of segments and performs text transcription (310). In some embodiments, the segments are divided based on pauses in the audio stream, which may correspond to sentences. In this example, the portion of the audio stream is segmented into "Hey, you." and "That *woman* is *f*cked".

[0065] In some embodiments, the audio machine learning model performs keyword-based and segment-based analysis on segments. Keyword-based analysis uses grammatical rules to identify harmful audio based on discrete word analysis. Keyword-based analysis may miss some instances of harmfulness because analyzing discrete words and not the segment context implies that those words must be objectively harmful regardless of their context. For example, the word "butt" may be used appropriately to refer to a part of the body, or inappropriately as an insult. Segment-based analysis identifies words that are harmful based on the context of the entire segment.

[0066] The audio machine learning model uses a first classifier, such as a first segment classifier, to generate a first segment label for each pair of segments (315). In this example, "Hi dog." is associated with a harmless label, and "That wh*re is f*cked" is associated with a harmful label. Specifically, the harmful labels relate to bullying, sexuality, and blasphemy. Because a broader context exists in the segment analysis, the harmful labels for the segments include additional bullying labels that were not identifiable from the keyword analysis.

[0067] Other labels may be used. For example, a harmful label may include one or more of the following: bullying and harassment, real-world dangerous activities, discrimination and hate, blackmail and intimidation, sexual content, violent content and bloodshed, violent threats, illegal and regulated, dating and romance, blasphemy, spam, political content, misleading impersonation or misrepresentation, destructive audio, fraud, and exploitation.

[0068] Segment labels may not capture all harmfulness. In some embodiments, the audio machine learning module 206 may generate keyword labels for each pair of segments to identify harmfulness missed by the segment labels (320). In this example, "Hi," "dog," "that," and "is" are associated with harmless labels, "wh*re" is associated with harmful label for sexual activity, and "f*cked" is associated with harmful label for blasphemy. Pairs of segments, segment labels, and keyword labels are added to the training dataset.

[0069] Referring to Figure 3B, another example 350 of the process for creating labels for audio segments is shown. A segment of the audio stream is received (355). The audio machine learning module 206 uses a first classifier to generate a first segment label (360). Harmful labels in this step are shown in bold. The audio machine learning module 206 retains the audio segment that has one or more harmful segments (365). The audio machine learning module 306 uses a second classifier, such as a second segment-level classifier, to generate a second segment label (370).

[0070] A second classifier may identify instances of abuse more accurately than the second segment label, but the second classifier may require more time to process the input (e.g., due to more complex detection techniques that require computational resources), and generating the output is computationally expensive. For example, the first part of an audio stream was identified by the first classifier as having only one harmful segment at 360, while the more accurate second classifier identified the first part of the audio stream as having two harmful segments at 370. The segment pairs and the second segment labels are added to the training dataset.

[0071] In some embodiments, the training data used for the audio classifier model includes audio streams collected with user permission for training and labeled by a human reviewer (e.g., a moderator). For example, the human reviewer listens to the audio streams in the training data, identifies whether each audio stream contains abuse, and, if so, times-stamps the location in the audio stream where the abuse occurs. The human-generated data is called the ground truth label. Such training data is then used to train the audio classifier model, for example, the audio classifier model during training generates a label for each audio stream in the training data, this label is compared to the ground truth label, and one or more model parameters are updated using a feedback function based on that comparison. In some embodiments, the human reviewer reviews the audio streams submitted as part of the abuse report provided by the user. The human review is conducted securely and confidentially, and the reviewer is specifically granted access to the audio streams for review and moderation. No user identification information is revealed to the reviewer.

[0072] The audio machine learning model is trained to receive portions of an audio stream as input. In some embodiments, the audio classifier model is applied iteratively to the audio stream, with each iteration corresponding to a portion of the audio stream, such as every 10 seconds, every 3 seconds, every minute, as additional audio is received.

[0073] Referring to Figure 3C, the third example, 375, demonstrates how an audio stream may be split by the audio machine learning module 206. Non-overlapping portions of the audio (i.e., speech chunks) are analyzed by an audio machine learning model trained on a dataset of segments containing labels for different types of audio. In this example, a 37-second portion of the audio stream is split into three audio portions: a 15-second portion (i1), a 15-second portion (i2), and a 7-second portion (i3). The maximum model output is compared to the three segment outputs across windows (i1, i2, i3, etc.). As a result, the audio stream is analyzed within segments rather than once across the entire stream.

[0074] In some embodiments, an audio machine learning model includes multiple classifiers. Referring to Figure 4, an exemplary block diagram of architecture 400 is shown, which includes a keyword classifier 405, a background noise classifier 410, and a segment-level classifier 415. The audio stream 402 is received by the keyword classifier 405, the background noise classifier 410, and the segment-level classifier 415. The keyword classifier 405 outputs labels for keywords, which are compared to a policy 420, and harmful words are identified based on the policy 420. Feedback is provided in real time (e.g., 0.5 seconds, 1 second, etc.). The goal of real-time feedback is to prevent harmful behavior before it occurs.

[0075] The background noise classifier 410 outputs the identification of background noise as problematic, and the background noise may be distracting, harmful, etc. Background noise may include speech effects and spoken words. The output of the background noise classifier 410 is compared with policy 420. Feedback is provided in near real time (e.g., 5 seconds, 7 seconds). The goal of near real time feedback is to proactively detect harmful behavior and proactively moderate the audio, for example, without receiving abuse reports from the user or other user behavior indicating potential abuse.

[0076] In some embodiments, the segment-level classifier 415 is a larger model than the keyword classifier 405, which processes one or more segments within a segment, and includes keyword and background noise analysis. In some embodiments, the segment-level classifier 415 combines information from other classifiers to enhance the analysis. The output is compared to policy 420, and feedback is also provided in delayed real time. In some embodiments, each classifier is stored on a separate server.

[0077] In some embodiments, the segment-level classifier 415 includes a deep neural network, such as a convolutional neural network. The deep neural network uses multiple layers to gradually extract higher-level features from the raw input, where the input to the layer is of a different type than the features extracted from other modules, and the output is a decision on whether or not the audio stream contains abuse.

[0078] The harmful categories may include blasphemy, bullying, dating and sexual behavior, racism, and others, where others is a mixture of categories with smaller amounts of training data, such as grooming, drugs and alcohol, self-harm, and radicalization. In some embodiments, the audio machine learning model generates synthetic audio to be used as training data. The synthetic audio may be used to generate training data for some of the harmful categories to address the lack of examples of training data. In some embodiments, synthetic audio and non-human voice audio are used to train the audio machine learning model.

[0079] Figure 5 is an illustrative figure 500 of how voice chat audio 505 is converted into synthesized audio in several embodiments. The voice chat audio 505 may include voice chat from various participants to a virtual experience generated by the moderation application 130 shown in Figure 1.

[0080] The voice chat audio 505 may be provided to both the automatic speech recognition system 510 and the voice kroner 515. The automatic speech recognition system 510 outputs the transcribed audio and provides the transcribed audio as input to the Large Language Model (LLM) 512, along with prompts specifying new text characteristics, such as a variant of the transcribed speech. The LLM 512 outputs the new text. For example, the transcribed speech may be an example of racism, such as "Go back to your country," and the LLM 512 outputs a variant of the racism based on the transcribed speech.

[0081] LLM may prompt you using the following list of speech moderation categories: “Bullying”: Speech in which the speaker bullies, stalks, trolls, or makes personal threats. “Blasphemy”: Speech that contains blasphemy. “Dating and Sex”: Speech that describes or suggests romantic or sexual activities involving the speaker and / or others. “Racism”: Speech that demeans others of a different race, ethnicity, or sexual orientation than the speaker. “No Violation”: Speech that does not fall under the categories of “Bullying,” “Blasphemy,” “Racism,” or “Dating and Sex.”

[0082] These instructions may also be: "Rewrite the examples as a single '(target)' sentence using words and phrases that teenagers might use, in the voice of the first or second person. Your answer should be a sentence without additional annotations."

[0083] The voice kroner 515 preserves the speaker characteristics of the original speaker and outputs an audio token. Both the new text and the audio token are provided as input to the text-speech system 520, which outputs synthesized audio 525. The audio token is used to preserve the characteristics of the original speaker and guide the text-speech system 520 to synthesize the speech with the same voice. Without the audio token, the synthesized audio 525 may lack subtle differences in tone, and the audio machine learning module 206 may be trained only on the text and not on the intonation used to speak the harmful phrase.

[0084] In some embodiments, the audio machine learning model includes multiple machine learning models. Figure 6A shows an exemplary architecture of an audio machine learning model 600 including multiple machine learning models, according to some embodiments. The feature extractor 602 may include one or more convolutional neural networks with multiple layers or local feature extraction of audio features. The feature extractor 602 receives the audio stream as a file or waveform. The audio stream is divided into time windows 616 (for example, each up to 15 seconds).

[0085] The feature extractor 602 identifies the word in question from the input for hazard detection. In some embodiments, the feature extractor 602 employs the Mel Frequency Cepstral Coefficient (MFCC) feature extraction technique. In some embodiments, the use of MFCCs can reduce computational cost and latency, resulting in, for example, a 40% acceleration during inference compared to the use of a convolutional neural network alone. The feature extractor 602 provides the extracted audio features and audio to the encoder 605.

[0086] Encoder 605 may be a pre-trained convolutional neural network that learns speech prediction and diagnoses audio streams in pre-training using self-supervised learning. Speech prediction training may occur through masking, where the training data is paired with a masked version of the complete sentence from which random words have been removed, in order to train Encoder 605 to predict masked words in the training data.

[0087] In some embodiments, the encoder 605 includes a convolutional neural network encoder that encodes audio and transmits the encoded audio to a multi-label classification model 610 and an audio-to-keyword detection model 615.

[0088] The multi-label classification model 610 may be a transformer encoder that identifies encoded audio as containing different types of labels. The transformer encoder includes an attention mechanism that uses weights assigned to relevant parts, thereby instructing the encoding layers to focus on specific parts of the audio input by weighting them to those values. The transformer encoder may include 24 encoding layers, ranging from 12 encoding layers with 8 attention heads each.

[0089] In some embodiments, the multi-label classification model 610 is trained by determining the mask prediction loss based on the target label. In some embodiments, the multi-label classification model 610 is trained using connectionist temporal classification (CTC) / minimum word error rate (MWER) loss and / or cross-entropy (CE) loss.

[0090] The audio-to-keyword detection model 615 classifies words in the encoded audio as good or bad for each window 616. The audio-to-keyword detection model 615 operates directly on the encoded audio by comparing the audio to a predefined list of keywords that match the bad category. In some embodiments, the audio-to-keyword detection model 615 is trained using connectionist-time classification (CTC) loss.

[0091] In some embodiments, the overall loss for the multi-label classification model 610 and the audio-to-keyword detection model 615 (known as the multi-task learning (MTL) loss) is defined using the following formula:

[0092]

number

[0093] During the ceremony,

[0094]

number

[0095] λ is the MTL loss, and λ is a hyperparameter.

[0096]

number

[0097] This is the CE loss,

[0098]

number

[0099] This is the CTC loss. In some embodiments, the hyperparameter is set to 0.7.

[0100] Figure 6B shows another exemplary architecture of the audio machine learning model 625. In this example, audio is received by encoder 630, which outputs the encoded audio to transformer 635, which is trained using cross-entropy loss, where parameters C1-C5 for cross-entropy loss include blasphemy 640a, dating and sexual activity 640b, bullying 640c, racism 640d, other 640e, and no violation 640f. The category other 640e may encompass a diverse range of harmful speech, including grooming, drugs and alcohol, radicalization, and references to other concepts that do not fit into the first four harmful categories. Encoder 630 may be trained using training data with labels corresponding to parameters C1-C5. In various embodiments, fewer or more categories may be used.

[0101] A trained audio machine learning model may be deployed on a virtual environment platform for detecting abuse within an audio stream, for example, on audio chats between users of the virtual environment platform. The audio machine learning model receives a portion of the audio stream (audio chat between users) as input. The audio machine learning model outputs an abuse decision. The abuse decision may be associated with an abuse score that reflects the level of abuse (e.g., mildly malicious, somewhat malicious, extremely malicious, etc.) and a confidence score that reflects the confidence level within the abuse score.

[0102] The abuse detection module 208 implements corrective actions based on the abuse determination. For example, the first time an abuse determination occurs, the corrective action may be a warning, and if an abuse determination occurs a second time, the corrective action may be a prohibition, such as muting the microphone on the client device 110 associated with the user who committed the abuse, preventing the audio stream associated with the user from contacting other users, or making the audio stream difficult to hear. In some embodiments, the abuse detection module 208 determines the corrective action based on an abuse score, where an abuse score exceeding an abuse threshold results in a more severe corrective action. In some embodiments, the abuse determination is associated with a confidence score, and the user's microphone is muted until the confidence score meets a confidence threshold. In some embodiments, other factors, such as the user's past history, are used to determine the corrective action.

[0103] Figure 6C shows an exemplary architecture of the cross-modal audio machine learning model 650. The audio machine learning model 650 includes an audio encoder 655 (also called the audio encoder) and a text encoder 664. The audio encoder 655 receives an audio input, and the text encoder 664 receives a text input, which is a transcribed version of the audio input. For example, the text input may be received from the audio stream shown in Figure 3A.

[0104] The audio encoder 655 encodes the audio and outputs an audio embedding 660. The audio embedding 660 is compared to a labeled, embedded audio to find its nearest neighbor. Once a nearest neighbor is found, a multi-label output 662 corresponding to the audio embedding 660 is determined, such as blasphemy, dating and sexual, bullying, racism, no violation, or any other label. This comparison is advantageous because the audio embedding 660 is multi-dimensional in the sense that it uses the tone from the audio input to determine how those words are used. For example, "What are you doing?" may sound friendly or accusatory depending on the context of the audio and the tone used within the audio.

[0105] The text system uses the corollary of audio tones to apply to the semantic meaning of the text. The text encoder 664 encodes the text and outputs text embeddings 666, which embed semantic meanings related to the text, for example, based on the entire grammar of the sentence. For example, "Today is a good day" and "The weather is nice" have similar semantic meanings.

[0106] Audio embeddings 660 and text embeddings 666 represent similar data (the same meaning expressed in both text and audio) across different modalities. Audio embeddings 660 and text embeddings 666 are compared, and the differences between them are reflected by a loss function, such as by using mean squared error loss 668. One or more parameters of the audio encoder 655 are adjusted to reduce the value of the loss function.

[0107] Figure 6D shows another exemplary architecture of the cross-modal audio machine learning model 675, including the toxicity classification portion 677 and the text injection portion 679.

[0108] The audio encoder 680 includes a convolutional neural network (CNN) feature extractor 676 and multiple transformer layers 678 (e.g., transformer layer 1 678a to transformer layer n 678n). The CNN feature extractor 676 extracts audio features from the audio input. The transformer layers 678 iteratively generate audio embeddings, each audio embedding corresponding to an audio token that mixes information from other input audio tokens via a self-attention mechanism. For example, each word in the audio input may be associated with an audio embedding.

[0109] The self-attention mechanism includes multiple attention heads that use attention scores and weights for relevant parts, thereby weighting those values ​​to instruct the transformer layer 678 how to focus on specific parts of the audio input. In some embodiments, the selected transformer layer 678 is quantized to reduce the model size. Quantization works by representing full-precision (32-bit) model weights with fewer bits. Quantization achieves both a reduction in model size and improved energy efficiency.

[0110] In some embodiments, the self-attention layer applies independent linear transformations to each audio embedding to generate query vectors, key vectors, and value vectors. The transformer layer 678 projects the audio embeddings, and each projection 682 carries its own set of learnable parameters, allowing the self-attention layer to focus on different semantic aspects within the sequence.

[0111] The projection 682 is provided as input to pooling 684. The goal of pooling 684 is to generate a single embedding vector representing the portion of the audio input received by the audio encoder 680. Pooling 684 combines individual audio embeddings to create a single embedding vector. For example, average pooling averages audio embeddings to create a single audio embedding. The audio embeddings are compared to labeled, embedded audio to find the nearest neighbor to the audio embedding. Once the nearest neighbor is found, the speech hazard classification 686 is determined.

[0112] The toxicity classification part 677 is the binary cross-entropy loss function (L BCE The model is trained to minimize the loss function using a binary cross-entropy loss function. In some embodiments, a single audio clip may contain multiple types of violations. In such cases, the hazard classification section 677 may perform multi-label classification, for example, by assigning multiple labels corresponding to different types of violations.

[0113] The text injection section 679 is applied at different layers of the network to demonstrate that a robust linear projection of the embedding space improves the accuracy of the audio machine learning model 675.

[0114] The text injection section 679 includes a text encoder 688 that receives a transcribed text, which is a transcription of the audio input. The text encoder 688 is trained on a variety of text content to produce rich semantic text embeddings corresponding to text tokens. The text embeddings of the audio transcription are useful representations that are potential candidates to be injected into the toxicology classification 677 training process to enhance semantic understanding by the speech audio encoder 680. The text embeddings are pooled 690, and a single text embedding 692 is output.

[0115] The transformer layer 678 projects audio embeddings, with each projection 694 carrying its own set of learnable parameters, and pooling 696 is used to aggregate the audio embeddings into a single audio embedding 698. The audio embedding 698 should be similar to the text embedding 692, since both of these embeddings are based on the same input. The text embedding 692 is a useful representation used during the training of the toxicity classifier 677 to enhance the semantic understanding of the speech by the audio encoder 680. The toxicity classifier 677 is trained by comparing the audio embedding 698 to the text embedding 692 to determine the loss function value. The parameters of the audio machine learning model are revised to reduce the value of the loss function.

[0116] In some embodiments, the text injection portion 679 is a linear combination of two losses, namely the classifier loss (L classifier ) and text injection loss values ​​(L) with more hyperparameters α text This is formalized as follows: In some embodiments, higher hyperparameters (e.g., 0.1 to 0.9) yield the best results. The classifier loss is calculated for the prediction of toxicity classification 477, and the text injection loss forces the text encoder output to match 480. The combined loss (L combined ) is defined as follows:

[0117] Lcombined=αLclassifier+(1-α)Ltext Formula 2

[0118] In some embodiments, binary cross-entropy loss is favorable for multi-label formulations and is therefore used for L-classifiers.

[0119] In some embodiments, mean squared error (MSE) loss and multi-class N-pair contrast loss are calculated. The MSE loss may be applied to a cross-modal encoder. The multi-class N-pair contrast loss may be applied to cross-modal training of speech and text machine translation. These losses may be applied in one of the transformer layers 678 within the audio encoder 680. If the target audio encoder layer i is a function li, the speech is s, and the corresponding transcription of the audio is t, then the two variables for their losses are as follows:

[0120] x = MeanPool(hProj(li(s))) Equation 3

[0121] y = MeanPool(t) Equation 4

[0122] In the formula, hProj is a learnable projection layer applied to the audio encoder layer output to reconcile the mismatched dimensionality between the layer output and the text encoder 488 output. The mean squared error for the text injection loss is calculated as follows:

[0123] L MSE =||xy||2 Equation 5

[0124] The control loss is calculated for each speech segment s and their corresponding transcription y within a training batch of N examples, where N-1 sets of transcriptions are selected.

[0125]

number

[0126] During the ceremony,

[0127]

number

[0128] is the set of all transcriptions in a given training batch, τ is the temperature hyperparameter, and sim is the cosine similarity function.

[0129]

number

[0130] In some embodiments, the control loss is most effective in classifying audio as harmful, particularly in the categories of dating and sexual activity, as well as racism and bullying.

[0131] The audio machine learning model may be trained to predict toxicity in multiple languages. In some embodiments, toxicity classifiers 677 and text injections 679 are trained in multiple languages, such as training both toxicity classifiers 677 and text injections 679 in the same language, and then training toxicity classifiers 677 and text injections 679 in the same additional languages ​​until the audio machine learning model is trained for all specified languages.

[0132] Exemplary Architecture Figure 7 shows an exemplary architecture 700 for using an audio machine learning model on portions of an audio stream to moderate the audio stream (for example, to detect malicious content). Portions of the audio stream are exported to storage 710 (705), and audio metadata is fetched locally (720). Local fetch (720) receives manifests and audio access from media access service 715. Local fetch (720) also receives open encoded audio from short-term cloud storage 725.

[0133] Local fetch (720) uses a message queuing service to fetch (745) the audio portion written to cloud storage (735). In addition, locally fetched information is stored in WAV format 730 and includes the audio written to cloud storage (735). If the information includes audio from the user, the audio is collected only upon receipt of user consent, and the audio is temporarily stored for training and removed once training is complete. Long-term cloud storage 740 stores audio machine learning models, audio embeddings, text embeddings, etc.

[0134] The audio portion is sent to the first speech recognition engine 755 by the message queuing service 750. The first speech recognition engine 755 performs speech recognition as discussed with reference to Figure 3B, and the second speech recognition engine 760 performs speech recognition on a subset of the audio portion. The labeling engine 765 labels the transcribed text, and the labels are stored in the long-term cloud storage 740.

[0135] Exemplary Moderation Process Figure 8 shows an exemplary process 800 for moderating an audio stream. When user 805 speaks (generating an audio stream sent to the platform), real-time detection 810 is performed on the audio stream by a moderation application 130 that outputs an abuse determination. If the abuse determination is a violation, the abuse detection module 208 may nudge or mute the user for a short period, for example, less than 5 seconds.

[0136] If an audio stream triggers an abuse report from another user 805, it may trigger an abuse reporting automation 820. In some embodiments, as a result of the submission of an abuse report 815, the abuse detection module 208 may impose a warning or ban for less than 5 minutes. For example, if the abuse detection module 208 determines that three sexual and blasphemous abuses have occurred and another user has submitted an abuse report, the user may be blocked from the virtual environment for 24 hours.

[0137] In some embodiments, the audio stream and abuse report are sent to human moderation 830. The human moderator reviews the audio stream and may impose a warning or ban of less than one hour. In some embodiments, the human moderator also labels the audio stream with different harmful labels, which are then used as ground truth data for training an audio machine learning model.

[0138] The abuse detection module 208 determines corrective actions to take against a user based on the output of an audio machine learning model that determines the user has committed an abuse. For example, if the abuse detection module 208 identifies three abuses within five minutes, it blocks the user from speaking for 24 hours. In another example, if the abuse detection module 208 determines that the user has committed abuses in multiple categories within 24 hours of being previously muted or blocked for abuse, the user's account will be permanently banned.

[0139] In some embodiments, the abuse detection module 208 applies different sets of rules based on the age of the first player. In some embodiments, the rules differ for the following groups: 13-16 years old, 16-18 years old, or 19 years old and over. For example, if the first user is 18 years old or older, the consequences of the first user committing abuse become more severe sooner than if the first user were a minor. Other age groups and other demographic factors (e.g., gender, sexual orientation, location, platform participation history, etc.) may be used as additions or substitutions.

[0140] The abuse detection module 208 receives a request to report an abuse. In some embodiments, the abuse detection module 208 sends the audio stream identified in the abuse report to a moderator, which reviews the audio and makes a determination as to whether an abuse occurred.

[0141] In some embodiments, the abuse detection module 208 may alert the first user in response to certain user signals, such as when the first user is silenced by two or more second users within 24 hours, or based on the acquisition of recommendations from an audio classifier model. In some embodiments, the abuse detection module 208 implements stronger corrective actions in response to the use of certain words.

[0142] The abuse detection module 208 may issue a series of warnings in response to user signals. For example, within a specific time window, such as a two-hour window, the abuse detection module 208 may issue up to four warnings with a cooling-off period of at least one minute between each warning. In some embodiments, the abuse detection module 208 may instruct the user interface module 202 to generate a first type of warning occupying a small portion of the screen for the first two instances, and a second type of warning occupying a larger portion of the screen and requesting an affirmative response from the first user for the third and fourth instances.

[0143] In some embodiments, the abuse detection module 208 may warn the first user when it first determines that the user is engaging in abuse before imposing corrective action.

[0144] Exemplary User Interface Figure 9 shows an exemplary user interface 900 that warns a first user that the word may result in corrective action. In this example, the user interface 900 includes a general warning that the first user has used word that violates the community standards, and a link 902 to the community standards. Clicking the link 902 may take the user to a different page that includes all of the community standards.

[0145] The user interface 900 also includes an agreement button 905 that indicates the first user acknowledges the warning. In some embodiments, the user interface includes a list of exemplary words (not shown) uttered by the first user that violate community standards. If the first user does not agree to the warning, the first user may click a disagree button 910 on the text, "Did I do something wrong? Let me know." In some embodiments, the abuse detection module 208 tracks the percentage of times the first user has selected the agreement button 905 over the disagree button 910.

[0146] The corrective action may take several forms and may be based on whether the first user is associated with a previous corrective action. In some embodiments, the abuse detection module 208 may impose a temporary ban for the first offense and a permanent ban for more serious offenses and / or repeated offenses. The ban may be an audio ban that prevents the first user from accessing its audio while still being able to experience the virtual experience, or a player ban that prevents the first user from experiencing the virtual experience for a certain period of time. For example, a player ban may include disabling login credentials.

[0147] In some embodiments, when the first user receives corrective action for the first time, the abuse detection module 208 may impose a first temporary ban (e.g., a one-day ban). When the first user receives corrective action for the second time, the abuse detection module 208 may impose a second temporary ban, which is longer than the first temporary ban (e.g., a three-day ban). When the first user receives corrective action for the third time, the abuse detection module 208 may impose a permanent ban. In some embodiments, the permanent ban is permanent in the sense that the length of the ban is indeterminate, but the permanent ban may be lifted by the abuse detection module 208 based on other factors (e.g., in response to a successful appeal by the first user against the permanent ban).

[0148] In some embodiments, after imposing corrective action, the user interface module 202 generates graphical data for displaying a user interface that includes a description of the abuse that led to the corrective action.

[0149] Exemplary method Figure 10 is a flowchart illustrating an exemplary method for generating training data for an audio machine learning model, according to several embodiments described herein. Method 1000 may be carried out by the computing device 200 in Figure 2.

[0150] Method 1000 may begin with block 1002, in which a user-provided audio stream related to the user is received. In some embodiments, prior to receiving the user-provided audio stream, Method 1000 further includes filtering the user-provided audio stream using a speech section detection (VAD) model to remove portions of the audio stream that do not contain human speech. In some embodiments, Method 1000 further includes filtering the user-provided audio stream to remove background noise. Block 1004 may follow block 1002.

[0151] In block 1004, the user-provided audio stream is divided into multiple parts, each part corresponding to a specific time window of the audio stream. Block 1006 may follow block 1004.

[0152] In block 1006, multiple portions of the user-provided audio stream are provided as input to the audio machine learning model. Block 1008 may follow block 1006.

[0153] In block 1008, the audio machine learning model outputs a determination of abuse for a specific portion of a user-provided audio stream, based on the portion of the stream. The abuse determination may include identification of the type of abuse, one or more groups of abuse such as blasphemy, bullying, harassment, and sexism, and combinations thereof. Block 1010 may follow block 1008.

[0154] In block 1010, corrective actions are taken in response to a determination of abuse of a particular part. The corrective actions may include giving a warning to the user. The corrective actions may include at least one of the following: muting the microphone on the user device associated with the user, or preventing the user-provided audio stream from being delivered to one or more other users.

[0155] Figure 11 is a flowchart illustrating an exemplary method for training an audio machine learning model to identify harmful elements in an audio stream, according to several embodiments described herein. Method 1100 may be carried out by the computing device 200 shown in Figure 2.

[0156] Method 1100 may begin with block 1102. In block 1102, a training audio stream of one or more people speaking is received. Block 1104 may follow block 1102.

[0157] In block 1104, for each training audio stream, the training audio stream is divided into two or more audio segments, the two or more audio segments are transcribed into two or more text segments, the first classifier generates a first segment label for each of the two or more text segments, the first segment label indicating whether the text segment is harmful or harmless. In some embodiments, when the first segment label indicates that the text segment is harmful, the first segment label further indicates a category selected from the groups of bullying, blasphemy, racism, harassment, and sexism, or a combination thereof. Block 1106 may follow block 1104.

[0158] In block 1106, a training audio stream, two or more text segments, and corresponding first segment labels from the training audio stream are added to the training dataset, and an audio machine learning model is trained using the training dataset to identify abuses of candidate training audio streams.

[0159] In some embodiments, Method 1100 further includes the steps of: identifying a subset of the training audio stream from a training audio stream, wherein one or more of a first segment labels indicate that one or more of the text segments are malicious; generating a second segment label for the subset of the training audio stream using a second classifier, wherein the second classifier identifies instances of abuse more accurately than the first classifier; and adding the second segment label to a training set. In some embodiments, Method 1100 further includes the step of generating a keyword label for each word in two or more text segments, wherein the two or more text segments and the corresponding first segment labels added to the training dataset further include the keyword label.

[0160] In some embodiments, Method 1100 further includes the steps of generating synthetic training audio by providing voice chat audio to an automated speech recognition (ASR) system; having the ASR system output transcribed audio based on the voice chat audio; providing the transcribed audio and a prompt specifying new text characteristics to a large language model (LLM), configured to generate new text based on the prompt and the transcribed audio; providing the voice chat audio to a voice kroner that outputs audio tokens storing speaker characteristics in the voice chat audio; providing the new text and audio tokens as input to a text-speech system; and having the text-speech system output synthetic training. In some embodiments, the new text characteristics correspond to one or more selected from the groups racism, grooming, drugs, alcohol, self-harm, and radicalization, or combinations thereof.

[0161] Figure 12 is a flowchart illustrating an exemplary method for moderating an audio stream according to several embodiments described herein. Method 1200 may be implemented by the computing device 200 shown in Figure 2.

[0162] Method 1200 may begin with block 1202, in which an audio input is provided to an audio encoder. Block 1204 may follow block 1202.

[0163] In block 1204, the audio encoder outputs an audio embedding corresponding to the audio input, and an audio hazard classification that identifies one or more hazardous labels to associate with the audio input, based on the audio input. Block 1204 may be followed by block 1206.

[0164] In block 1206, a text input is provided to a text encoder, and the text input is a transcription of the audio input. Block 1208 may follow block 1206.

[0165] In block 1208, the text encoder outputs text embeddings based on the text input. Block 1210 may follow block 1208.

[0166] In block 1210, the value of the text injection loss function is determined based on a comparison of audio and text embeddings. Block 1212 may follow block 1210.

[0167] In block 1212, one or more parameters of the audio encoder are adjusted to reduce the value of the text injection loss function.

[0168] In some embodiments, the audio input includes real-world audio related to abuse reports from one or more users, and Method 1200 further includes the steps of comparing the audio toxicity classification to labels associated with the real-world audio in order to determine the value of the classifier loss function, wherein the labels associated with the real-world audio are ground truth provided by human judges, and adjusting the parameters of the audio encoder to reduce the value of the classifier loss function. In some embodiments, the classifier loss function is a binary cross-entropy loss, and the text injection loss function is a mean squared error (MSE) loss. In some embodiments, Method 1200 further includes the step of generating a combined loss function based on a linear combination of the classifier loss function and the text injection loss function with hyperparameters. In some embodiments, the audio encoder is trained using a control loss for each speech segment and a transcription for each speech segment.

[0169] The methods, blocks, and / or operations described herein may be performed in an order different from that shown or described, and / or may be performed (partially or completely) concurrently with other blocks or operations as appropriate. Some blocks or operations may be performed on a portion of the data and then later, for example, on another portion of the data. Note that in various embodiments, not all of the described blocks and operations may be performed. In some embodiments, blocks and operations may be performed multiple times, in different orders, and / or in different ways at different points in time.

[0170] The various embodiments described herein include steps of acquiring data from various sensors in a physical environment, analyzing such data, generating recommendations, and providing a user interface. Data acquisition is performed only with specific user permission and in accordance with applicable regulations. Data is stored in accordance with applicable regulations, including anonymizing the data to protect user privacy, or, in some cases, modifying it. Clear information regarding data collection, storage, and use is provided, and the user is given the option to select the types of data that may be collected, stored, and used. Furthermore, the user controls the devices on which data may be stored (e.g., client device 110 only, client + server device, etc.) and the devices on which data is analyzed (e.g., client device 110 only, client + server device, etc.). Data is used for specific purposes as described herein. No data is shared with third parties without explicit user permission.

[0171] In the above description, numerous specific details are given for illustrative purposes and to provide a complete understanding of this specification. However, it will become apparent to those skilled in the art that this disclosure can be practiced without these specific details. In some cases, structures and devices are shown in block diagram form to avoid obscuring the description. For example, embodiments may be described above primarily with reference to user interfaces and specific hardware. However, embodiments may be applied to any type of computing device capable of receiving data and commands, and any peripheral device that provides services.

[0172] Any reference in this specification to “some embodiments” or “some examples” means that certain features, structures, or characteristics described in relation to the embodiments or examples may be included in at least one embodiment of this specification. The phrase “in some embodiments” appearing in various places in this specification does not necessarily refer to the same embodiment.

[0173] Some parts of the detailed description above are presented in terms of algorithms and symbolic representations of operations on data bits in computer memory. These descriptions and presentations of algorithms are the means used by those skilled in data processing to communicate their work most effectively to others skilled in the art. An algorithm should here be considered, generally speaking, as a set of self-consistent steps leading to a desired result. These steps are those that require the physical manipulation of physical quantities. These quantities, though not always, take the form of electrical or magnetic data that can be stored, transferred, combined, compared, or, if applicable, manipulated. For primarily general use, it has sometimes proven convenient to refer to these data as bits, values, elements, symbols, characters, terms, numbers, etc.

[0174] However, it should be noted that all these and similar terms should be associated with appropriate physical quantities and are merely convenient labels applied to those quantities. As will be evident from the following discussion, unless otherwise specified, discussions using terms such as “process,” “calculate,” “calculate,” “determine,” or “display” will be understood to refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities in the registers and memory of a computer system and converts it into other data similarly represented as physical quantities in the computer system memory or registers, or other storage devices, transmission devices, or display devices of such information.

[0175] Embodiments of this specification may also relate to a processor for performing one or more steps of the methods described above. The processor may be a dedicated processor selectively activated or reconfigured by a computer program stored in the computer. Such computer programs may be stored in non-temporary computer-readable storage media, including, but not limited to, any type of disk including optical disks, ROM, CD-ROM, magnetic disk, RAM, EPROM, EEPROM, magnetic card or optical card, flash memory including a USB key with non-volatile memory, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0176] This specification may take the form of some complete hardware embodiment, some complete software embodiment, or some embodiment that includes both hardware and software elements. In some embodiments, this specification is implemented in software, including, but not limited to, firmware, resident software, microcode, etc.

[0177] Furthermore, this specification may take the form of a computer program product accessible from a computer-enabled medium or computer-readable medium that provides program code for use by or with a computer or any instruction execution system. For the purposes of this specification, a computer-enabled medium or computer-readable medium may be any device capable of containing, storing, communicating, propagating, or transferring a program for use by or with an instruction execution system, apparatus, or device.

[0178] A data processing system suitable for storing or executing program code would include at least one processor directly or indirectly coupled to a memory element via a system bus. The memory element may include local memory used during the actual execution of the program code, bulk storage, and cache memory providing temporary storage for at least some program code to reduce the number of times the code must be retrieved from bulk storage during execution. [Explanation of symbols]

[0179] 100 Network Environment 102 Online Virtual Experience Server 104 Virtual Experience Engine 105 Virtual Experiences 108 Datastores 110 Client Devices 110a Client Device 110b client device 110n client device 112 Virtual Experience Applications 112a Virtual Experience Application 112b Virtual Experience Application 112n virtual experience application 114 I / O Interfaces 114a I / O Interface 114b I / O Interface 114n I / O interface 122 Network 125 users 130 Moderation Applications 200 computing devices 202 User Interface Module, Module 204 Speech Recognition Engine, Module 206 Audio Machine Learning Module, Module 208 misuse detection module, module 218 Bus 222 signal line 224 signal line 226 signal line 228 signal line 230 signal line 232 signal line 234 signal line 235 processors 237 memory 239 Input / Output (I / O) Interfaces 241 Microphone 243 speakers 245 displays 247 Storage Devices 300 Process Examples Another example of process 350 375 Third Example 400 Architecture 402 Audio Stream 405 Keyword Classifier 410 Background noise classifier 415 Segment-Level Classifier 420 Policy 488 Text Encoder 500 Figures 505 Voice Chat Audio 510 Automatic Speech Recognition System 512 Large-Scale Language Models (LLMs) 515 krona (voice) 520 Text-Speech System 525 Synthetic Audio 600 Audio Machine Learning Models 602 Feature Extractor 605 encoder 610 Multi-label classification models 615 Keyword detection model from audio 616 windows 625 Audio Machine Learning Models 630 encoders 635 Transformers 640a Blasphemy 640b Dating and Sex 640c bullying 640d racism 640e et al. 640f No violations 650 Cross-Modal Audio Machine Learning Models 655 Audio Encoder 660 Audio embedding 662 Multi-label output 664 Text Encoder 666 Text embedding 668 Mean Squared Error Loss 675 Cross-modal audio machine learning models, audio machine learning models 676 Convolutional Neural Network (CNN) Feature Extractor 677 Hazard classification part, Hazard classification 678 Transformer Layer 678a Transformer layer 1 678n Transformer layer n 679 Text injection section, injection 680 Audio Encoders 682 Projection 684 Pooling 686 Classification of Voice Harmfulness 688 Text Encoder 690 will be pooled. 692 Text embedding 694 Projection 696 Pooling 698 Audio embedding 700 Architecture 710 storage 715 Media Access Service 720 Fetch locally 725 Short-term cloud storage 730 WAV format 735 Write to cloud storage 740 Long-term cloud storage 750 Message Queuing Service 755 First speech recognition engine 760 Second speech recognition engine 765 Labeling Engine 800 processes 805 users 810 Real-time detection 815 Abuse Report 820 Automated abuse reporting 830 human moderation 900 User Interface 902 Link 905 Agree button 910 Disagree button 1000 ways 1100 methods 1200 methods

Claims

1. A computer implementation method for moderating an audio stream, The steps include receiving a user-provided audio stream related to the user, A step of dividing the user-provided audio stream into multiple parts, wherein each part corresponds to a specific time window of the audio stream, The steps include providing the user-provided audio stream to an audio machine learning model as input, The steps include: using the audio machine learning model to output a decision on the misuse of a specific part of the user-provided audio stream based on that part; In accordance with the aforementioned determination of misuse of the specific part, the steps include implementing corrective measures and Methods that include...

2. The aforementioned audio machine learning model, The steps include providing an audio input to an audio encoder, The steps include: using the audio encoder to output an audio hazard classification that identifies an audio embedding corresponding to the audio input and one or more hazardous labels to associate with the audio input; A step of providing text input to a text encoder, wherein the text input is a transcription of the audio input, The steps include: outputting text embedding based on the text input using the text encoder; The steps include determining the value of the text injection loss function based on a comparison of the audio embedding and the text embedding, The steps include adjusting one or more parameters of the audio encoder in order to reduce the value of the text injection loss function, The method according to claim 1, which is trained by

3. The audio input includes real-world audio related to abuse reports from one or more users, and the method is A step of determining the value of the classifier loss function, the audio toxicity classification is compared to a label associated with the real-world audio, wherein the label associated with the real-world audio is ground truth provided by a human judge. The steps include adjusting the parameters of the audio encoder in order to reduce the value of the classifier loss function, and The method according to claim 2, further comprising:

4. The aforementioned audio machine learning model is trained using training data, and the method is The steps include receiving a training audio stream of one or more people speaking, For each training audio stream, The steps include dividing the training audio stream into two or more audio segments, The steps include: transcribing the two or more audio segments into two or more text segments; A step of generating a first segment label for each of the two or more text segments using a first classifier, wherein the first segment label indicates whether the text segment is harmful or harmless. The steps include adding the training audio stream, the two or more text segments, and the corresponding first segment labels from the training audio stream to the training dataset. The step of generating the training data by The method according to claim 1, further comprising:

5. The step of generating the training data is, A step of identifying a subset of the training audio stream from the training audio stream, wherein one or more of the first segment labels indicate that one or more of the text segments are harmful. A step of generating a second segment label for the subset of the training audio stream using a second classifier, wherein the second classifier identifies instances of abuse more accurately than the first classifier. The steps include adding the second segment label to the training set and The method according to claim 4, further comprising:

6. The aforementioned audio machine learning model is trained using synthetic training data, and the method is Steps include providing voice chat audio to an automated speech recognition (ASR) system, The ASR system provides a step of outputting transcribed audio based on the voice chat audio, A step of providing a large language model (LLM) with the transcribed audio and prompts specifying new text characteristics, wherein the LLM is configured to generate new text based on the prompts and the transcribed audio. The steps include providing the voice chat audio to a voice kroner that outputs an audio token that stores speaker characteristics within the voice chat audio, The steps include providing the new text and the audio token as input to a text-speech system, The text-speech system provides the synthesized training and The method according to claim 1, further comprising the step of generating the synthetic training data by means of the method.

7. The method according to claim 1, wherein the corrective action includes giving a warning to the user.

8. The method according to claim 1, wherein the corrective action includes at least one of muting a microphone on a user device associated with the user, or preventing the user-provided audio stream from being distributed to one or more other users.

9. The method according to claim 1, wherein the determination of abuse comprises identifying a type of abuse, the type of abuse being selected from one or more of the groups of blasphemy, bullying, harassment, and sexism, and combinations thereof.

10. A system for moderating audio streams, One or more processors, A memory connected to one or more processors, which stores instructions thereon. The instruction includes, and when the instruction is executed by the processor, the processor: Receiving user-provided audio streams related to the user, The process involves classifying the user-provided audio stream into multiple parts, wherein each part corresponds to a specific time window of the audio stream. The user-provided audio stream is provided to the audio machine learning model as input to the aforementioned multiple portions of the user-provided audio stream, The audio machine learning model outputs a decision on the misuse of a specific part of the user-provided audio stream based on the part of the user-provided audio stream, In accordance with the aforementioned determination of misuse of the specific part, corrective measures shall be implemented. A system that performs actions including those mentioned above.

11. The aforementioned audio machine learning model, Providing audio input to an audio encoder, The audio encoder outputs an audio hazard classification that identifies an audio embedding corresponding to the audio input and one or more hazardous labels to associate with the audio input, based on the audio input. Providing text input to a text encoder, wherein the text input is a transcription of the audio input, The text encoder outputs text embedding based on the text input, The value of the text injection loss function is determined based on a comparison of the audio embedding and the text embedding. To reduce the value of the text injection loss function, one or more parameters of the audio encoder are adjusted. The system according to claim 10, which is trained by

12. The aforementioned audio input includes real-world audio related to abuse reports from one or more users, and the aforementioned operation is, To determine the value of the classifier loss function, the audio toxicity classification is compared to a label associated with the real-world audio, wherein the label associated with the real-world audio is ground truth provided by a human judge. To reduce the aforementioned value of the classifier loss function, the parameters of the audio encoder are adjusted. The system according to claim 11, further comprising:

13. The aforementioned audio machine learning model is trained using training data, and the operation is as follows: Receiving a training audio stream of one or more people speaking, For each training audio stream, The training audio stream is divided into two or more audio segments, Transcribing the two or more audio segments into two or more text segments, Using a first classifier, generate a first segment label for each of the two or more text segments, wherein the first segment label indicates whether the text segment is harmful or harmless. Adding the training audio stream, the two or more text segments, and the corresponding first segment labels from the training audio stream to the training dataset. The system according to claim 10, further comprising generating the training data by the method described above.

14. The training data mentioned above is generated as described above. Identifying a subset of the training audio stream from the training audio stream, wherein one or more of the first segment labels indicate that one or more of the text segments are harmful. Using a second classifier, generate a second segment label for the subset of the training audio stream, wherein the second classifier identifies and generates instances of abuse more accurately than the first classifier. Adding the second segment label to the training set The system according to claim 13, further comprising:

15. The aforementioned audio machine learning model is trained using synthetic training data, and the method is Providing voice chat audio to an automated speech recognition (ASR) system, The ASR system outputs transcribed audio based on the voice chat audio, Providing a large-scale language model (LLM) with the transcribed audio and prompts specifying new text characteristics, wherein the LLM is configured to generate new text based on the prompts and the transcribed audio. The voice chat audio is provided to a voice kroner that outputs an audio token that stores the speaker characteristics within the voice chat audio. The new text and the audio token are provided as input to the text-speech system, The text-speech system outputs the synthesized training. The system according to claim 10, further comprising generating the synthetic training data by means of the method.

16. A non-temporary computer-readable medium on which instructions are stored, wherein when an instruction is executed by one or more computers, the one or more computers are made to perform an action, and the action is performed Receiving user-provided audio streams related to the user, The user-provided audio stream is divided into multiple parts, each part corresponding to a specific time window of the audio stream. The user-provided audio stream is provided to the audio machine learning model as input to the aforementioned multiple portions of the user-provided audio stream, The audio machine learning model outputs a decision on the misuse of a specific part of the user-provided audio stream based on the part of the user-provided audio stream, In accordance with the aforementioned determination of misuse of the specific part, corrective measures shall be implemented. Non-temporary computer-readable media, including [specific examples of such media].

17. The aforementioned audio machine learning model, Providing audio input to an audio encoder, The audio encoder outputs an audio hazard classification that identifies an audio embedding corresponding to the audio input and one or more hazardous labels to associate with the audio input, based on the audio input. Providing text input to a text encoder, wherein the text input is a transcription of the audio input, The text encoder outputs text embedding based on the text input, The value of the text injection loss function is determined based on a comparison of the audio embedding and the text embedding. To reduce the value of the text injection loss function, one or more parameters of the audio encoder are adjusted. A non-temporary computer-readable medium according to claim 16, which is trained by the following:

18. The aforementioned audio input includes real-world audio related to abuse reports from one or more users, and the aforementioned operation is, To determine the value of the classifier loss function, the audio toxicity classification is compared to a label associated with the real-world audio, wherein the label associated with the real-world audio is ground truth provided by a human judge. To reduce the aforementioned value of the classifier loss function, the parameters of the audio encoder are adjusted. A non-temporary computer-readable medium according to claim 17, further comprising:

19. The aforementioned audio machine learning model is trained using training data, and the operation is as follows: Receiving a training audio stream of one or more people speaking, For each training audio stream, The training audio stream is divided into two or more audio segments, Transcribing the two or more audio segments into two or more text segments, Using a first classifier, generate a first segment label for each of the two or more text segments, wherein the first segment label indicates whether the text segment is harmful or harmless. Adding the training audio stream, the two or more text segments, and the corresponding first segment labels from the training audio stream to the training dataset. The non-temporary computer-readable medium according to claim 16, further comprising generating the training data by means of the method.

20. The non-temporary computer-readable medium according to claim 16, wherein the corrective action includes providing a warning to the user.

Images