Artificial delay for managing voice communications

Abstract

A computer-implemented method for determining whether to introduce a delay in an audio stream from a particular speaker includes an audio stream from a transmitting device. The method further includes providing the audio stream and the speech analysis score, information about the one or more sound emotion parameters, and one or more sound emotion scores of a first user associated with the transmitting device as inputs to a trained machine learning model, the trained machine learning model is iteratively applied to the audio stream, and each iteration corresponds to a respective portion of the audio stream. The method further includes generating, as an output, a degree of detrimental of the audio stream using the trained machine learning model. The method further includes transmitting the audio stream to a receiving device, where the transmitting is performed to introduce a time delay in the audio stream based on the degree of detrimental.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is an international application and claims priority under 35 U.S.C. §119(e) to U.S. patent application No. 17 / 940,749, filed on September 8, 2022, and entitled “ARTIFICIAL LATENCY FOR MODERATING VOICE COMMUNICATION,” the entire contents of which are incorporated herein by reference. Background Art

[0002] Online platforms need a way to provide a safe and civil environment for communication between user devices. Text communication is easier to manage than audio communication because users are more tolerant of delays in the transmission of text messages. In addition, it is easier to manage text communication than audio streams because the text can be compared to a list of banned words or problem words. In contrast, audio streams are more difficult to manage because of variations in accent, intonation, volume, use of sarcasm, etc.

[0003] The background description provided herein is intended to introduce the background of the present disclosure. The work done by the current inventor, to the extent described in this background section, and in all aspects of the specification that may not constitute prior art at the time of filing, shall not be deemed to be prior art for the present disclosure, whether expressly or impliedly. Summary of the invention

[0004] Embodiments generally relate to a system and method for introducing artificial delays into an audio stream for management. According to one aspect, a computer-implemented method includes receiving an audio stream from a sending device. The method also includes providing the audio stream and a speech analysis score, information about one or more voice emotion parameters, and one or more voice emotion scores of a first user associated with the sending device as input to a trained machine learning model, wherein the trained machine learning model is iteratively applied to the audio stream, and wherein each iteration corresponds to a corresponding portion of the audio stream. The method also includes generating a degree of harmfulness of the audio stream as an output using the trained machine learning model. The method also includes sending the audio stream to a receiving device, wherein the sending is performed to introduce a time delay in the audio stream based on the degree of harmfulness.

[0005] In some embodiments, the method further includes identifying harmful instances in the audio stream, and replacing the harmful instances in the audio stream with noise or silence before sending the audio stream to the receiving device. In some embodiments, the method further includes identifying silence or pauses between words in the audio stream, the silence or pauses corresponding to specific timestamps in the audio stream, wherein a time delay is introduced at the specific timestamps of silence or pauses between words as a gap in the audio stream. In some embodiments, the method further includes updating the speech analysis score based on identifying harmful instances in the audio stream. In some embodiments, the method further includes receiving text from a text channel associated with the sending device, wherein the text channel is separated from the audio stream, and generating a text score indicating the harmful level of the text, wherein the input of the trained machine learning model also includes the text score. In some embodiments, the input of the trained machine learning model also includes the harmful history of the first user, the speaker history and metadata associated with the first user, and the listener history and metadata associated with the second user, the second user being associated with the receiving device. In some embodiments, the one or more voice emotion parameters include tone, pitch, and voicing intensity level determined based on one or more previous audio streams from the sending device. In some embodiments, the audio stream is provided with the visual signal, and the method further comprises synchronizing the visual signal with the audio stream by introducing a time delay in the visual signal that is the same as the time delay of the audio stream. In some embodiments, the audio stream is part of the video stream, and the method further comprises analyzing the audio stream to identify the harmful instance, in response to identifying the harmful instance, detecting a portion of the video stream depicting an offensive gesture, wherein the offensive gesture occurs within a predetermined time period of the harmful instance, and in response to detecting the offensive gesture, modifying at least the portion of the video stream by one or more of: blurring the portion, or replacing the portion with pixels that match a background area. In some embodiments, the audio stream is part of the video stream, and the method further comprises performing motion detection on the video stream to detect the offensive gesture, and in response to detecting the offensive behavior, modifying at least the portion of the video stream by one or more of blurring the portion or replacing the portion with pixels that match a background area. In some embodiments, if the harmfulness is below a minimum threshold, the time delay is zero seconds.

[0006] According to one aspect, a device includes a processor and a memory coupled to the processor, the memory having instructions stored thereon, which, when executed by the processor, cause the processor to perform operations, the operations including: receiving an audio stream from a sending device, providing the audio stream and a speech analysis score, information about one or more sound emotion parameters, and one or more sound emotion scores of a first user associated with the sending device as inputs to a trained machine learning model, wherein the trained machine learning model is iteratively applied to the audio stream, and wherein each iteration corresponds to a corresponding portion of the audio stream, and generating a degree of harmfulness of the audio stream as an output using the trained machine learning model; and sending the audio stream to a receiving device, wherein the sending is performed to introduce a time delay in the audio stream based on the degree of harmfulness.

[0007] In some embodiments, the operations further include identifying harmful instances in the audio stream and replacing the harmful instances in the audio stream with noise or silence before sending the audio stream to the receiving device. In some embodiments, the operations further include identifying silence or pauses between words in the audio stream, the silence or pauses corresponding to specific timestamps in the audio stream, wherein a time delay is introduced at the specific timestamps of the silence or pauses between words as a gap in the audio stream. In some embodiments, the operations further include updating the speech analysis score based on identifying harmful instances in the audio stream.

[0008] According to one aspect, a non-transitory computer-readable medium stores thereon instructions which, when executed by one or more computers, cause the one or more computers to perform operations, including: receiving an audio stream from a sending device, providing the audio stream and a speech analysis score, information about one or more sound emotion parameters, and one or more sound emotion scores of a first user associated with the sending device as inputs to a trained machine learning model, wherein the trained machine learning model is iteratively applied to the audio stream, and wherein each iteration corresponds to a corresponding portion of the audio stream, generating a degree of harmfulness of the audio stream as an output using the trained machine learning model, and sending the audio stream to a receiving device, wherein the sending is performed to introduce a time delay in the audio stream based on the degree of harmfulness.

[0009] In some embodiments, the above operations also include identifying harmful instances in the audio stream and replacing the harmful instances in the audio stream with noise or silence before sending the audio stream to the receiving device. In some embodiments, the above operations also include identifying silence or pauses between words in the audio stream, the silence or pauses corresponding to specific timestamps in the audio stream, wherein a time delay is introduced at the specific timestamps of the silence or pauses between words as a gap in the audio stream. In some embodiments, the above operations also include updating the speech analysis score based on identifying harmful instances in the audio stream. In some embodiments, the above operations also include receiving text from a text channel associated with the sending device, wherein the text channel is separated from the audio stream, and generating a text score indicating the harmfulness level of the text, wherein the input of the trained machine learning model also includes the text score.

[0010] One approach to preventing harmful content from appearing in audio streams is to buffer the audio stream, identify harmful instances before the audio stream is sent from the sending device to the receiving device, and remove them from the audio stream. However, managing the audio stream can introduce multiple seconds of latency into the audio stream. Audio latency of more than 50 milliseconds can cause unnatural delays that disrupt conversations, while latency of more than 250 milliseconds can cause conversations to break down.

[0011] The present application advantageously describes a Metaverse engine and / or Metaverse application that provides a method for identifying harmful instances while selectively inserting gaps or pauses in an audio stream so that management can be performed without perceptible delays. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram of an example network environment for identifying harmful instances in communications according to some embodiments described herein.

[0013] Figure 2 is a block diagram of an example computing device for identifying harmful instances in communications according to some embodiments described herein.

[0014] Figure 3 is an example user interface for a video stream in which harmful instances are identified according to some embodiments described herein.

[0015] Figure 4 is an example user interface for a video stream in which offensive behavior is identified according to some embodiments described herein.

[0016] Figure 5 is an example flow chart for identifying harmful instances in communications according to some embodiments described herein.

[0017] Figure 6is another example flow chart for identifying harmful instances in communications according to some embodiments described herein. DETAILED DESCRIPTION

[0018] Network environment 100

[0019] Figure 1 A block diagram of an example environment 100 for identifying harmful instances in communications is shown. In some embodiments, the environment 100 includes a server 101, user devices 115a, ..., 115n, and a network 105. Users 125a, ..., 125n can be associated with respective user devices 115a, ..., 115n. Figure 1 In the remaining figures, a letter following a reference number, such as "115a", indicates a reference to the element with that specific reference number. A reference number without a following letter in the text, such as "115", indicates a general reference to various embodiments of the element with that reference number. In some embodiments, environment 100 may include Figure 1 Other servers or devices not shown in the figure. For example, the server 101 can be multiple servers 101.

[0020] Server 101 includes one or more servers, each server including a processor, a memory, and network communication hardware. In some embodiments, server 101 is a hardware server. Server 101 is communicatively coupled to network 105. In some embodiments, server 101 sends data to user device 115 and receives data from user device 115. Server 101 may include metaverse engine 103 and database 199.

[0021] In some embodiments, the Metaverse Engine 103 includes code and programming for receiving communications between two or more users in a virtual Metaverse, such as between friends at the same location in the Metaverse, within the same Metaverse experience, or within a Metaverse application. Users interact across different demographics (e.g., different ages, regions, languages, etc.) within the Metaverse.

[0022] In some embodiments, the metaverse engine 103 receives an audio stream from a user device 115a, the destination of which is a user device 115n. The metaverse engine 103 provides the audio stream and the speech analysis score, information about one or more voice emotion parameters, and one or more voice emotion scores of the user 115a associated with the user device 115a as input to the trained machine learning model. The trained machine learning model is iteratively applied to corresponding portions of the audio stream, such as a few seconds when the audio stream is received.

[0023] The Metaverse Engine 103 generates as an output a degree of harmfulness of the audio stream using the trained machine learning model. The Metaverse Engine 103 transmits the audio stream to one or more other user devices 115n, wherein the transmission is performed to introduce a time delay in the audio stream based on the degree of harmfulness. In some embodiments, the Metaverse Engine 103 uses the time delay to identify harmful instances and replaces the harmful instances with noise or silence before transmitting the audio stream to one or more other user devices 115n.

[0024] In some embodiments, the Metaverse Engine 103 is implemented using hardware including a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), any other type of processor, or a combination thereof. In some embodiments, the Metaverse Engine 103 is implemented using a combination of hardware and software.

[0025] Database 199 can be a non-transitory computer-readable memory (e.g., random access memory), a cache, a drive (e.g., a hard drive), a flash drive, a database system, or another type of component or device capable of storing data. Database 199 can also include multiple storage components (e.g., multiple drives or multiple databases) that can also span multiple computing devices (e.g., multiple server computers). Database 199 can store data associated with the Metaverse Engine 103, such as training data sets for trained machine learning models, history and metadata associated with each user 125, and the like.

[0026] User device 115 may be a computing device including memory, a hardware processor, and a camera. For example, user device 115 may include a mobile device, a tablet computer, a mobile phone, a wearable device, a head mounted display, a mobile email device, a portable game console, a portable music player, a reader device, or other electronic device capable of accessing network 105 and capturing images with a camera.

[0027] User device 115a includes Metaverse application 104a, and user device 115n includes Metaverse application 104b. In some embodiments, user device 115a is a sending device and user device 115n is a receiving device. In some embodiments, user 125a generates a communication, such as an audio stream or a video stream, using Metaverse application 104a on the sending device, and the communication is sent to Metaverse engine 103. Once the communication is approved for sending, Metaverse engine 103 sends the communication to Metaverse application 104b on the receiving device for access by user 125n.

[0028] In the illustrated embodiment, the entities of environment 100 are communicatively coupled via network 105. Network 105 may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wired network (e.g., Ethernet), a wireless network (e.g., an 802.11 network, A network, or a wireless LAN (WLAN), a cellular network (e.g., a long term evolution (LTE) network), a router, a hub, a switch, a server computer, or a combination thereof. Figure 1 One network 105 is shown coupled to the server 101 and the user device 115, but in practice one or more networks 105 may be coupled to these entities.

[0029] Computing device example 200

[0030] Figure 2 1 is a block diagram of an example computing device 200 that can be used to implement one or more features described herein. Computing device 200 can be any suitable computer system, server, or other electronic or hardware device. In some embodiments, computing device 200 is server 101. In some embodiments, computing device 200 is user device 115.

[0031] In some embodiments, computing device 200 includes processor 235, memory 237, input / output (I / O) interface 239, microphone 241, speaker 243, display 245, and storage device 247. Depending on whether computing device 200 is server 101 or user device 115, some components of computing device 200 may not be present. For example, if computing device 200 is server 101, computing device 200 may not include microphone 241 and speaker 243. In some embodiments, computing device 200 includes Figure 2 Additional components not shown.

[0032] The processor 235 can be coupled to the bus 218 via signal line 222, the memory 237 can be coupled to the bus 218 via signal line 224, the I / O interface 239 can be coupled to the bus 218 via signal line 226, the microphone 241 can be coupled to the bus 218 via signal line 228, the speaker 243 can be coupled to the bus 218 via signal line 230, the display 245 can be coupled to the bus 218 via signal line 232, and the storage device 247 can be coupled to the bus 218 via signal line 234.

[0033] Processor 235 includes an arithmetic logic unit, a microprocessor, a general purpose controller, or some other processor array to perform calculations and provide instructions to the display device. Processor 235 processes data and may include a variety of computing architectures, including a complex instruction set computer (CISC) architecture, a reduced instruction set computer (RISC) architecture, or an architecture that implements a combination of instruction sets. Although Figure 2 A single processor 235 is shown, but multiple processors 235 may be included. In different embodiments, the processor 235 may be a single-core processor or a multi-core processor. Other processors (e.g., a graphics processing unit), operating system, sensors, display, and / or physical configuration may be part of the computing device 200.

[0034] Memory 237 stores instructions and / or data that can be executed by processor 235. Instructions may include code and / or programs for performing the techniques described herein. Memory 237 may be a dynamic random access memory (DRAM) device, a static RAM, or some other memory device. In some embodiments, memory 237 also includes non-volatile memory (e.g., a static random access memory (SRAM) device or flash memory), or similar permanent storage devices and media (including hard drives, compact disc read only memory (CD-ROM) devices, DVD-ROM devices, DVD-RAM devices, DVD-RW devices, flash memory devices), or some other large-capacity storage devices for more permanent storage of information. Memory 237 includes code and programs that are operable to execute the Metaverse Engine 103, which will be described in more detail below.

[0035] The I / O interface 239 may provide functionality that enables the computing device 200 to interface with other systems and devices. The interface device may be included as part of the computing device 200 or may be independent and communicate with the computing device 200. For example, network communication devices, storage devices (e.g., memory 237 and / or storage device 247), and input / output devices may communicate via the I / O interface 239. In another example, the I / O interface 239 may receive data from the server 101 and deliver the data to the metaverse engine 103 and components of the metaverse engine 103, such as the machine learning module 210. In some embodiments, the I / O interface 239 may be connected to an interface device, such as an input device (keyboard, pointing device, touch screen, microphone 241, sensor, etc.) and / or an output device (display device, speaker 243, monitor, etc.).

[0036] Some examples of interface devices that can be connected to I / O interface 239 can include display 245, which can be used to display content (e.g., images, videos, and / or user interfaces of output applications as described herein) and to receive touch (or gesture) input from a user. Display 245 can include any suitable display device, such as a liquid crystal display (LCD), a light emitting diode (LED), or a plasma display screen, a cathode ray tube (CRT), a television, a monitor, a touch screen, a three-dimensional display screen, or other visual display device.

[0037] Microphone 241 includes hardware for detecting audio spoken by a person. Microphone 241 can send audio to Metaverse Engine 103 via I / O interface 239.

[0038] Speaker 243 includes hardware for generating audio for playback. For example, speaker 243 receives instructions from Metaverse Engine 103 and generates audio from another user after determining that the audio stream does not include harmful instances. Speaker 233 converts the instructions into audio and generates audio for the user.

[0039] Storage device 247 stores data related to Metaverse engine 103. For example, storage device 247 may store training data sets for trained machine learning models, history and metadata associated with each user 125, etc. In an embodiment where computing device 200 is server 101, storage device 247 and Metaverse engine 103 may store data related to Metaverse engine 103. Figure 1 The same as database 199 in.

[0040] Example Metaverse Engine 103 or Metaverse Application 104

[0041] Figure 2 A computing device 200 is shown executing an example Metaverse engine 103 or Metaverse application 104, the computing device 200 including a history module 202, a speech analyzer 204, a voice emotion analyzer 206, a text module 208, a machine learning module 210, a toxicity module 212, and a user interface module 214. Although the modules are shown as part of the same Metaverse engine 103 or Metaverse application 104, one of ordinary skill in the art will recognize that the above modules may be implemented by the computing device 200. For example, the text module 208 may be part of a user device 115, the text module 208 providing analysis of text communications before the text communications are sent to the Metaverse engine 103 as part of the server 101, so as to reduce the computing requirements of the server 101.

[0042] The history module 202 generates historical information and metadata about users participating in the communication. In some embodiments, the history module 202 includes an instruction set that can be executed by the processor 235 to generate historical information and metadata. In some embodiments, the history module 202 is stored in the memory 237 of the computing device 200 and can be accessed and executed by the processor 235.

[0043] In some embodiments, after obtaining user permission, the history module 202 stores information about each communication session in the metaverse associated with the user and metadata associated with the user. Communication sessions may include audio streams, video streams, text communications, etc. After obtaining user permission, the history module 202 may store information about harmful instances associated with the user. For example, the history module 202 may identify when a user participates in a harmful instance, the harmful behavior they perform (e.g., swearing, making offensive behavior, bullying other users, threatening other users, etc.), identify specific users targeted by harmful instances, etc. In all cases where information about a user is stored, the history module 202 has obtained permission from the user, the user has been informed that the user can delete the information, and the information is securely stored and complies with applicable regulations. Further details are discussed below with reference to the user interface module 214.

[0044] In some embodiments, after obtaining user permission, history module 202 can store information about the context of harmful instances, such as specific experiences. For example, a user may use a lot of profanity when playing a violent shooting game, but may not exhibit toxic behavior in a non-violent role-playing game. History module 202 can receive information about harmful instances from other modules, such as speech analyzer 204 and harmful module 212.

[0045] In some embodiments, after obtaining user permission, the history module 202 stores listener history and metadata about the user's reaction to another user's implementation of harmful instances against him. For example, the history module 202 can update the listening history and metadata to indicate whether the user is insensitive to insulting language, whether the user responds to insulting language with the same insulting language, whether the user reports the user after seeing offensive behavior, etc. In another example, the history module 202 can track the number of times a particular user has been blocked and whether the particular user participated in an event where other users who blocked the particular user were also present. In some embodiments, the history module 202 generates a sensitivity score that uses a scale (e.g., 3 out of 10, 0.9 out of 1, etc.) to reflect the user's sensitivity to harmful instances.

[0046] In some embodiments, after obtaining the user's permission, the history module 202 stores metadata associated with the user. For example, the metadata may include the region where the user lives, other demographic information (gender, gender identity, age, race, preferred pronouns, sexual orientation, etc.), one or more Internet Protocol (IP) addresses associated with the user, one or more languages ​​spoken by the user, etc. In some embodiments, the history module 202 may characterize the user's response in conjunction with the metadata. For example, the history module 202 may identify that the user is generally insensitive to harmful instances in the metaverse unless the user is subjected to insulting words related to their religious beliefs, gender identity, race, etc.

[0047] The history module 202 can provide the history information and metadata as input to the trained machine learning model to the machine learning module 210. The history module 202 can also provide the history information and metadata to other modules to provide context that affects the calculation of harmful instances. For example, the speech analyzer 204 receives the history information and metadata because the speech analyzer 204 uses different rules to identify harmful instances for users aged 13-16, 16-18, or over 18.

[0048] In some embodiments, speech analyzer 204 analyzes speech during a communication session. In some embodiments, speech analyzer 204 includes an instruction set that is executable by processor 235 to analyze speech during a communication session. In some embodiments, speech analyzer 204 is stored in memory 237 of computing device 200 and can be accessed and executed by processor 235.

[0049] The speech analyzer 204 receives the audio stream from the sending device. Because speech analysis may take several seconds, the speech analyzer 204 performs a continuous analysis of the speech in the audio stream. The analysis may be retrospective, i.e., the speech analyzer 204 performs the analysis after the audio stream has been sent to the receiving device, or the analysis may be performed each time an unwanted instance is identified, regardless of whether the audio stream has been sent to the receiving device.

[0050] In some embodiments, speech analyzer 204 comprises a machine learning model that is trained to predict various properties of an audio stream, such as vocal effort, speaking style, language, spoken activity, and the like.

[0051] The speech analyzer 204 can perform automatic speech recognition (ASR), such as speech-to-text translation, and compare the translated text to the harmful word list to identify harmful instances in the audio stream. The speech analyzer 204 can generate a speech analysis score for a user associated with the sending device. For example, the speech analyzer 204 can generate a speech analysis score based on identifying harmful instances associated with the audio stream.

[0052] In some embodiments, the speech analyzer 204 generates a speech analysis score based on demographic information of a particular user. For example, based on whether the user is 13-16, 16-18, 18+ (or 12-15, 15-18, 18+, etc.), based on the user's location, based on whether the audio stream is sent to users with different demographic information (e.g., when the audio stream is sent to a 13-year-old user, the audio stream may be identified as including a harmful instance), or based on the type of game (e.g., a shooter game vs. a puzzle game), the speech analyzer 204 applies different criteria for what constitutes a harmful instance.

[0053] In some embodiments, the speech analyzer 204 performs speech-to-text translation based on the one or more languages ​​spoken by the user. For example, the aquatic mammal "seal" in English is called "phoque" in French, and "phoque" is not confused with a harmful instance (i.e., the vulgar word "fuck" in English). In some embodiments, the identification of the one or more languages ​​spoken by the user is received as part of the metadata determined by the history module 202.

[0054] In some embodiments, the speech analyzer 204 periodically provides speech analysis (e.g., speech analysis scores) to the machine learning module 210 as input to the trained machine learning model. The speech analysis scores can be associated with timestamps so that the speech analysis scores are aligned with positions in the audio stream. In some embodiments, the speech analyzer 204 sends the speech analysis scores to the machine learning module 210 whenever the speech analyzer 204 identifies an unwanted instance in the audio stream and updates the speech analysis scores reflecting the identified unwanted instance.

[0055] The voice emotion analyzer 206 analyzes the emotion in the audio stream. In some embodiments, the voice emotion analyzer 206 includes an instruction set that can be executed by the processor 235 to analyze the emotion in the audio stream. In some embodiments, the voice emotion analyzer 206 is stored in the memory 237 of the computing device 200 and can be accessed and executed by the processor 235.

[0056] In some embodiments, the voice emotion analyzer 206 identifies different speakers in the audio stream and associates the different speakers with corresponding audio identifiers. The voice emotion analyzer 206 analyzes multiple sound parameters, such as one or more of the following: tone, pitch, and vocal intensity level of each user in the audio stream. In some embodiments, the voice emotion analyzer 206 analyzes the tone by determining the positivity and energy of the user's voice. For example, the voice emotion analyzer 206 detects whether the user sounds excited, annoyed, neutral, sad, etc. In some embodiments, the voice emotion analyzer 206 determines the emotional state of the speaker based on the emotional quadrant. The emotional quadrant includes four states: nervous, happy, angry, and sad. The voice emotion analyzer 206 can detect emotions using a transformer-based technique, such as using wav2vec2.0 as part of the front end.

[0057] In some embodiments, the sound emotion analyzer 206 analyzes pitches ranging from 60 Hertz (Hz) to 2 kHz. In some embodiments, the sound emotion analyzer 206 determines the fundamental frequency of the sound and determines the range of pitches present in the audio stream.

[0058] In some embodiments, the voice emotion analyzer 206 analyzes the vocal intensity level by determining the noise level and comparing the level to a predetermined vocal intensity description. For example, the voice emotion analyzer 206 can determine that a person whispering produces a vocal intensity of 20-30 decibels (dB), a person speaking softly produces a vocal intensity of 30-55 dB, a person speaking at an average level produces a vocal intensity of 55-65 dB, a person speaking loudly or yelling produces a vocal intensity of 65-80 dB, and a person screaming produces a vocal intensity of 80-120 dB. In some embodiments, the voice emotion analyzer 206 also identifies whether the vocal intensity level increases over time, as this may indicate that the conversation is escalating into an argument that may cause harassment.

[0059] In some embodiments, the voice emotion analyzer 206 generates a voice emotion score for the user associated with the audio stream. In some embodiments, the voice emotion analyzer 206 generates a separate score for each of the following: tone, pitch, and vocal intensity level determined based on one or more previous audio streams from the sending device. In some embodiments, the voice emotion analyzer 206 generates a voice emotion score that is a combination of tone, pitch, and vocal intensity. For example, because a user does not shout when angry, it may not be obvious whether the user is angry unless there is a combination of angry tone, large pitch changes, and low vocal intensity levels.

[0060] Regardless of whether the user uses voice modulation software, the voice emotion analyzer 206 can analyze the emotions in the audio stream. For example, when the user selects voice modulation software to make him sound like a popular cartoon character, the voice emotion analyzer 206 detects that the voice modulation is taking place and performs emotion analysis regardless of how the voice is modulated.

[0061] The voice emotion analyzer 206 may periodically generate one or more voice emotion scores and send information about the voice emotion parameters and the one or more voice emotion scores to the machine learning module 210. For example, each time the speech analyzer 204 identifies an objectionable instance in the audio stream and updates the speech analysis score to reflect the identified objectionable instance, the voice emotion analyzer 206 sends information about the voice emotion parameters and the one or more voice emotion scores to the machine learning module 210. In another example, each time the voice emotion parameters change, such as when the tone, pitch, or vocal intensity level changes, or when the change in tone, pitch, or vocal intensity level exceeds one or more predetermined thresholds (e.g., when the user changes from a vocal intensity level associated with normal speech to a vocal intensity level associated with shouting), the voice emotion analyzer 206 sends information about the voice emotion parameters and the one or more voice analysis scores.

[0062] In some embodiments, the voice emotion analyzer 206 can be stored on the sending device and perform a quick analysis of the voice emotion. For example, the sending device may include a voice emotion analyzer 206 with a 60% accuracy rate for tone detection. The voice emotion analyzer 206 on the sending device can send information to the voice emotion analyzer 206 on the server 101, which performs a more detailed analysis of the voice emotion.

[0063] The text module 208 analyzes the text from the text channel. In some embodiments, the text module 208 includes an instruction set that can be executed by the processor 235 to analyze the text. In some embodiments, the text module 208 is stored in the memory 237 of the computing device 200 and can be accessed and executed by the processor 235.

[0064] In some embodiments, a user can simultaneously participate in an audio stream and send text messages using a separate text channel. For example, during a video call, a user may be giving a presentation and also adding additional information to a chat box associated with the video call. In another example, a user may participate in a game via an audio stream while sending direct text messages to a specific user through the game software. Analyzing the audio stream and text messages may be helpful because some users may be polite in the audio stream while being abusive to a specific member during the game by sending a private message.

[0065] In some embodiments, the text module 208 compares the text to the harmful word list and identifies harmful instances in the text. The text module 208 can generate a text score indicating the harmfulness level of the text based on the text message. The text module 208 can periodically generate the text score and send the text score to the machine learning module 210. In some embodiments, the text module 208 sends the text score whenever a harmful instance in the text is identified and the text module 208 updates the text score accordingly.

[0066] In some embodiments, the text module 208 removes harmful instances from the text before sending the text to the receiving device. The text module 208 can simply remove harmful instances, such as a first user threatening another user. Or the text module 208 can include a warning and an explanation of the reason for removing harmful instances from the text.

[0067] In some embodiments, the text module 208 is stored on the sending device as part of the Metaverse application 104a and analyzes the text on the sending device to save computing resources of the server 101. The text module 208 on the sending device can send the text score to the Metaverse engine 103 as input to the machine learning module 210.

[0068] The machine learning module 210 trains a machine learning model (or multiple models) to output the harmfulness of the audio stream. In some embodiments, the machine learning module 210 includes an instruction set that can be executed by the processor 235 to train the machine learning model to output the harmfulness of the audio stream. In some embodiments, the machine learning module 210 is stored in the memory 237 of the computing device 200 and can be accessed and executed by the processor 235.

[0069] In some embodiments, the machine learning module 210 obtains a training data set having an audio stream that is manually labeled and paired with outputs from one or more of the following: the history module 202, the speech analyzer 204, the sound emotion analyzer 206, and the text module 208. In some embodiments, the manual labels include harmful instances in the audio stream and meta information, such as language, emotional state, vocal intensity, etc. For example, the training data set may include an audio stream paired with the following outputs: output from the history module 202, including one or more of the following: a harmful history of a first user (i.e., a speaker), a speaker history and metadata associated with the first user, and a listener history and metadata associated with a second user; output from the speech analyzer 204, including a speech analysis score for the first user that is sent periodically; output from the sound emotion analyzer 206, including a speech analysis score for the first user that is sent periodically, and in some embodiments, a separate speech analysis score including tone, pitch, and vocal intensity level; and output from the text module 208, including a text score.

[0070] In some embodiments, the training data set also includes automatically labeled audio streams that have been pre-processed for offline harmfulness detection. The labels may indicate timestamps of where harmful instances occur in the audio stream.

[0071] In some embodiments, the training data set also includes a synthesized audio stream of the speech-to-text translation, wherein the synthesized audio stream includes a large corpus of harmful speech and harmless speech. The synthesized audio stream can be labeled as including harmful speech and harmless speech, and can include timestamps detailing where the harmful instances occurred.

[0072] In some embodiments, the training data set also includes audio streams enhanced with different parameters to help train the machine learning model to output the harmfulness of the audio stream in different situations. For example, the training data set can be enhanced with an audio stream that includes varying pitches of speakers in the audio stream, noise, codecs, echoes, background distractors (e.g., traffic, natural sounds, people who are muffled, etc.), music, and playback speed.

[0073] The machine learning module 210 trains the machine learning model using the training data set in a supervised learning manner. The training data set includes audio stream examples without harmful content and audio stream examples with one or more harmful instances, which enables the machine learning module 210 to use the distinction between harmful and harmless activities as labels in the supervised learning process to train the machine learning model to classify the input audio stream as including harmful activities or harmless activities.

[0074] In some embodiments, the training data for the machine learning model includes audio streams collected with user permission for training purposes and labeled by a human reviewer. For example, a human reviewer listens to the audio streams in the training data and identifies whether each audio stream includes harmful instances, and if so, timestamps the location within the audio stream where the harmful instance occurs. The artificially generated data is referred to as groundtruth labels. The model is then trained using these training data, for example, the model being trained generates a label for each audio stream in the training data, compares it to the groundtruth label, and updates one or more model parameters using a feedback function based on the comparison result.

[0075] In some embodiments, the training data for the machine learning model also includes video streams. The training data set can be labeled to include examples of video streams without offensive behavior and examples of video streams with one or more offensive behaviors, which enables the machine learning module 210 to use the distinction between offensive and non-offensive behaviors as labels during supervised learning to train the machine learning model to classify input video streams as including offensive behavior or not offensive behavior. For example, a training data set with offensive behavior can include a set of actions corresponding to offensive behavior, such as lip movement to form a swear word, arm movement within a threshold time period to form an action associated with a swear word, etc., as well as actions that are precursors to offensive behavior, such as an arm starting to move in a specific way that may lead to offensive behavior. The video stream can include a video of the user or a video of the user's virtual character.

[0076] In some embodiments, the machine learning module 210 is a deep neural network. Types of deep neural networks include convolutional neural networks, deep belief networks, stacked autoencoders, generative adversarial networks, variational autoencoders, flow models, recurrent neural networks, and attention-based models. A deep neural network uses multiple layers to progressively extract higher-level features from the original input, where the input to the layer is different types of features extracted from other modules, and the output is a decision on whether to perform management.

[0077] The machine learning module 210 can generate layers that recognize increasingly detailed features and patterns within the speech of the audio stream, where the output of one layer is used as input to a subsequent more detailed layer until the final output is the harmfulness of the audio stream. An example of different layers in a deep neural network can include token embeddings, segment embeddings, and positional embeddings.

[0078] In some embodiments, the machine learning module 210 trains the machine learning model using a back-propagation algorithm. The back-propagation algorithm modifies the internal weights of the input signal based on feedback (e.g., at each node / layer of a multi-layer neural network), which can be a function of the output labels produced by the model being trained (e.g., "this portion of the audio stream is level 1 harmful") and the true value labels included in the training data (e.g., "this portion of the audio stream is level 2 harmful"). This weight adjustment can improve the accuracy of the model being trained.

[0079] After the machine learning module 210 trains the machine learning model, the trained machine learning model receives the following inputs: a speech analysis score of a first user associated with a sending device from the speech analyzer 204, information of one or more voice emotion parameters and one or more voice emotion scores about the first user from the voice emotion analyzer 206, and an audio stream from the sending device. The information about the one or more voice emotion parameters may include information about the tone, pitch, and vocal intensity level of the user in the audio stream.

[0080] In some embodiments, the machine learning module 210 also receives the harmful history of the first user from the history module 202, the speaker history and metadata associated with the first user, and the listener history and metadata associated with the second user associated with the receiving device. Metadata can be used to identify whether the speaker may violate community guidelines. For example, a banned user may create a new user profile, but the metadata includes indicators indicating that it is the same user, such as an IP address and crowd information. In some embodiments, the machine learning module 210 also receives a text score indicating the harmful level of the text from the text module 208. In some embodiments, the trained machine learning model periodically receives a speech analysis score, information about one or more sound emotion parameters, and one or more sound emotion scores. In some embodiments, the trained machine learning model continuously receives an audio stream, and the trained machine learning model is iteratively applied to the audio stream, wherein each iteration corresponds to a corresponding portion of the audio stream. For example, the trained machine learning model can be applied to the audio stream every 1 second, every 2 seconds, every 0.5 seconds, etc.

[0081] The trained machine learning model generates as output a harmfulness rating of the audio stream, which is a reflection of how harmful the audio stream is currently and a prediction that the audio stream is likely to become more harmful.

[0082] In some embodiments, the machine learning module 210 sends the harmfulness level of the audio stream to the harmfulness module 212. The machine learning module 210 may provide the harmfulness level each time the trained machine learning model generates the harmfulness level.

[0083] The harmful module 212 introduces a time delay to the audio stream based on the degree of harmfulness and analyzes the audio stream for harmful instances. In some embodiments, the harmful module 212 includes an instruction set that is executable by the processor 235 to introduce the time delay and analyze the audio stream for harmful instances. In some embodiments, the user interface module 214 is stored in the memory 237 of the computing device 200 and can be accessed and executed by the processor 235.

[0084] In some embodiments, the harmfulness module 212 receives the harmfulness level of the audio stream from the machine learning module 210. The harmfulness level may correspond to the time delay of sending the audio stream to the receiving device. For example, a level of 0 may indicate that there is no possibility of harmfulness in the audio stream, and the audio stream may be sent without time delay.

[0085] In some embodiments, the harmful module 212 determines whether to introduce a time delay in the transmission audio stream so that there is enough time to analyze the audio stream for harmful instances. For example, the time delay can be 0 to 5 seconds. In some embodiments, if the harmfulness is below a minimum threshold, the time delay is zero seconds and the harmful module 212 sends the audio stream to the receiving device. In some embodiments, if the harmfulness exceeds the minimum threshold, the harmful module 212 determines the amount of delay to be applied based on the increasing harmfulness. In some embodiments, the harmful module 212 also applies a stricter level of scrutiny when the harmfulness indicates that the audio stream is more likely to be harmful. Higher transmission delays are a negative feedback mechanism that inhibits harmful behavior by slowing down interactions or preventing effective communication even without additional management. In some embodiments, the harmfulness may be so high that the harmful module 212 mutes the user. For example, if the user uses profanity every other word, it may be easier to simply mute the audio stream until the user's aggressive speech ends.

[0086] In some embodiments, harmful module 212 performs speech-to-text translation of the audio stream, or receives a translation from one of the other modules. In some embodiments, harmful module 212 performs speech-to-text translation after the speaker has finished speaking a sentence. In some embodiments, harmful module 212 identifies harmful instances in an audio stream without first converting the audio stream to text.

[0087] The harmful module 212 identifies harmful instances in the audio stream. In some embodiments, the harmful module 212 gradually adjusts the time delay of the audio stream transmission to avoid creating perceptible audio distortion, and aims to adjust for gaps between words or sentences. For example, when silence or pauses are present, the harmful module 212 introduces gaps to make the gaps less noticeable. Pauses and silence can be detected in less than 100ms, so the time required for the harmful module 212 to identify pauses and silence is less than the time required to wait for a sentence to end. In some embodiments, the harmful module 212 tracks the timestamps of all audio signals / packets to help add gaps in the correct locations and make the transition between speech and silence more seamless.

[0088] In some embodiments, harmful module 212 replaces harmful instances in an audio stream with noise, or censors harmful instances by replacing them with silence.

[0089] In some embodiments, the audio stream is part of the visual signal. The visual signal can be an animation, such as an avatar animation or a physical animation, or the visual signal can be a video stream, wherein the audio stream is part of the video stream. The visual signal is presented to one or more other users, and the above users all participate in the metaverse, wherein the user's representation (e.g., their avatar) is in the same area of ​​the metaverse so that each avatar can see the other avatars when the users interact. In the event that the harmful module 212 introduces a time delay in the audio transmission, the harmful module 212 synchronizes the time delay with the video signal so that the visual signal has the same time delay as the audio stream.

[0090] In some embodiments, harmful module 212 analyzes the video stream for harmful instances. In response to harmful module 212 identifying harmful instances in the audio stream, harmful module 212 may analyze the video stream for offensive behavior that occurred within a predetermined time period of the harmful instance. Figure 3 , shows an example user interface 300 for a video stream in which the harmful module 212 identifies harmful instances in an audio stream. The harmful module 212 performs image recognition on the video stream of the user avatar and identifies locations within the video stream where the speaker's mouth moves to form words corresponding to the harmful instances in the audio stream. The harmful module 212 instructs the user interface module 214 to overlay a graphic 305 on the speaker's mouth when the speaker performs an offensive behavior. Because the graphic 305 draws attention to the offensive behavior, other mitigating actions may also be taken, such as adding a blur effect to the mouth, or even replacing the mouth with pixels that match the background.

[0091] In some embodiments, harmful module 212 performs motion detection and / or object detection on the video to identify offensive behavior. In response to identifying the offensive behavior, harmful module 212 blurs the offensive behavior in the video stream or replaces the offensive behavior with pixels that match the background. In some embodiments, harmful module 212 can analyze the user's virtual character while performing motion detection and / or object detection to determine whether the virtual character looks excited and use the virtual character as a signal that the user may be performing offensive behavior.

[0092] Go to Figure 4 , another example user interface 400 of a video stream is shown in which an offensive behavior is identified. In this example, the harmful module 212 determines that the user is about to commit an offensive behavior and that the user looks angry. The harmful module 212 generates a mask 405 that replaces pixels associated with the hand with pixels from the background to make the offensive behavior invisible.

[0093] The user interface module 214 generates a user interface. In some embodiments, the user interface module 214 includes an instruction set that can be executed by the processor 235 to generate a user interface. In some embodiments, the user interface module 214 is stored in the memory 237 of the computing device 200 and can be accessed and executed by the processor 235.

[0094] The user interface module 214 generates a user interface for a user 125 associated with the user device 115. The user interface may be used to initiate audio communications with other users, participate in games in the metaverse, send texts to other users, initiate video communications with other users, etc. In some embodiments, the user interface includes an option for adding user preferences, such as the ability to block other users 125.

[0095] In some embodiments, before a user participates in the Metaverse, the user interface module 214 generates a user interface that includes information about how user information is collected, stored, and analyzed. For example, the user interface requires the user to provide permission to use any information associated with the user. The user is informed that the user can delete user information and that the user has the option to select the type of information provided for different purposes. The use of information complies with applicable regulations and the data is stored securely. Data collection is not performed in certain locations or for certain user categories (e.g., based on age or other population information), data collection is temporary (i.e., the data is discarded after a period of time), and the data is not shared with third parties. Some data may be anonymized, aggregated across users, or otherwise modified so that the identity of a particular user cannot be determined.

[0096] In some embodiments, user interface module 214 provides a user interface that explains to the user that the metaverse engine 103 automatically detects harmful instances and may store audio or video of the harmful instances in association with the user's account.

[0097] Example Method

[0098] Figure 5 is an example flow chart 500 for identifying harmful instances in communications. In flow chart 500, bold lines represent data flows including audio flows, and thin lines represent data flows of information.

[0099] Flowchart 500 includes audio and text communications from a sending device 505 to a receiving device 510. The sending device 505 receives audio input via a microphone and performs analog-to-digital conversion of the audio stream, compresses the audio stream, and sends the audio stream to a real-time server 515. The real-time server 515 sends the audio stream to a constant-length multi-second buffer 520, which sends the audio stream to a module 525 that performs continuous retrospective speech analysis. The continuous retrospective speech analysis 525 is not performed in real time to allow sufficient time to improve the accuracy of the analysis. The real-time server 515 sends the audio stream to a stream selection / mute / noise module 555 to be forwarded to the receiving device 510 if the machine learning module 530 determines that the audio stream does not need to be delayed. If the machine learning module 530 determines that the audio stream needs to be delayed, the real-time server 515 sends the audio stream to an adjustable multi-second buffer 545.

[0100] The audio stream is also sent to a module 535 that performs voice emotion analysis. The voice emotion analysis is sent as input to the machine learning module 530.

[0101] The sending device 505 also receives text input via a keyboard. The sending device 505 performs text encoding and sends the text to a module that performs text management 540. The module that performs text management 540 sends the text to the receiving device 510 and sends the text as input to the machine learning module 530.

[0102] The machine learning module 530 also receives as input the harmful history of each game, the speaker's history and metadata, and the listener's history and metadata.

[0103] The machine learning module 530 predicts the likelihood that harmfulness is about to occur and determines the length of time to buffer the audio. If there is no likelihood of harmfulness occurring, there is no delay in the audio stream, and the machine learning module 530 sends the audio stream directly to the receiving device 510. If there is a likelihood of harmfulness occurring, the machine learning module 530 sends the audio stream to the adjustable multi-second buffer 545. The adjustable multi-second buffer 545 sends the audio stream to the module 550 that detects actual harmfulness. The module 550 that detects actual harmfulness sends the harmful instance to the module 555 that determines whether to select a stream, mute a stream, or add noise to a stream. The stream selector / mute / noise generator module 555 sends the audio stream to the receiving device 510.

[0104] Figure 6 is another example flow chart 600 for identifying harmful instances in communications according to some embodiments described herein. In some embodiments, the Metaverse engine 103 is stored on the server 101. In some embodiments, the Metaverse engine 103 is stored on the user device 115. In some embodiments, the Metaverse engine 103 is stored partially on the server 101 and partially on the user device 115.

[0105] Method 600 may begin at block 602. At block 602, an audio stream is received from a transmitting device. Block 602 may be followed by block 604.

[0106] At block 604, an input of a trained machine learning model is provided, the input including an audio stream and a speech analysis score, information about one or more voice emotion parameters, and one or more voice emotion scores of a first user associated with a transmitting device. The trained machine learning model is applied iteratively to portions of the audio stream, wherein each iteration corresponds to a respective portion of the audio stream. Block 604 may be followed by block 606.

[0107] At block 606 , the trained machine learning model generates as output a harmfulness level of the audio stream. Block 606 may be followed by block 608 .

[0108] The audio stream is transmitted to a receiving device at block 608. The transmitting is performed to introduce a time delay in the audio stream based on the degree of detrimentality.

[0109] The methods, frames, and / or operations described herein may be performed in a different order than shown or described, and / or may be performed simultaneously (partially or completely) with other frames or operations where appropriate. Some frames or operations may be performed on a portion of data, and later, for example, performed again on another portion of data. Not all described frames and operations need to be performed in various embodiments. In some embodiments, frames and operations may be performed multiple times in different orders and / or at different times in the method.

[0110] Various embodiments described herein include obtaining data from various sensors in a physical environment, analyzing such data, generating recommendations, and providing a user interface. Data collection is performed only with the permission of a specific user and in compliance with applicable regulations. The storage of data complies with applicable regulations, including anonymizing the data or otherwise modifying the data to protect user privacy. Provide users with clear information about data collection, storage, and use, and provide options to select the types of data that can be collected, stored, and used. In addition, users control the devices that can store data (e.g., only user devices, client and server devices, etc.) and the devices that perform data analysis (e.g., only user devices, client and server devices, etc.). Data is used for specific purposes described herein. No data will be shared with third parties without the explicit permission of the user.

[0111] In the above description, for the purpose of explanation, many specific details are listed to provide a thorough understanding of the specification. However, it is obvious to those skilled in the art that the present disclosure can be practiced without these specific details. In some cases, in order to avoid vague descriptions, structures and devices are shown in block diagram form. For example, each embodiment can be described mainly with reference to a user interface and specific hardware. However, each embodiment can be applied to any type of computing device that can receive data and commands, and any external device that provides services.

[0112] References to "some embodiments" or "some examples" in the specification mean that a particular feature, structure, or characteristic described in conjunction with the embodiments or examples may be included in at least one implementation of the specification. The phrase "in some embodiments" appearing in different places in the specification does not necessarily refer to the same embodiment.

[0113] Some portions of the above detailed description are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the art of data processing to most effectively convey the substance of their work to others skilled in the art. An algorithm is generally considered to be a series of self-consistent steps that lead to a desired result. These steps are steps that require physical manipulations of physical quantities. Typically, though not necessarily, these quantities are in the form of electrical or magnetic data that can be stored, transferred, combined, compared, and otherwise manipulated. It turns out that it is sometimes convenient to refer to these data, mainly for common reasons, as bits, values, elements, symbols, characters, terms, numbers, or similar names.

[0114] However, it should be noted that all of these terms and similar terms are associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless explicitly stated in the following discussion, throughout the description, discussions using the terms 'processing', 'computing', 'deriving', 'displaying', etc. refer to the actions and processes of a computer system or similar electronic computing device. These actions and processes involve the manipulation and conversion of data represented in the form of physical (electronic) quantities in the registers and memories of the computer system, and transforming them into other data also represented in the form of physical quantities, which are stored in the memory or registers of the computer system or other information storage, transmission, or display devices.

[0115] Embodiments of the present specification may also relate to a processor for performing one or more steps of the above method. The processor may be a dedicated processor selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory computer-readable storage medium, which includes but is not limited to any type of disk, including an optical disk, ROM, CD-ROM, magnetic disk, RAM, EPROM, EEPROM, magnetic card or optical card, flash memory including a USB key with non-volatile memory, or any type of medium suitable for storing electronic instructions, each of which is coupled to a computer system bus.

[0116] The specification can take the form of some completely hardware embodiments, some completely software embodiments, or some embodiments comprising hardware and software elements. In some embodiments, the specification is implemented in software, including but not limited to firmware, resident software, microcode, etc.

[0117] Additionally, the description may take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For purposes of the description, a computer-usable or computer-readable medium may be any device that can contain, store, communicate, propagate, or transport the program for use by or in connection with an instruction execution system, device, or apparatus.

[0118] A data processing system suitable for storing or executing program code will include at least one processor connected directly or indirectly to a memory element via a system bus. The memory element may include local memory, bulk storage, and cache memory used when actually executing the program code, which provides at least temporary storage of some program code to reduce the number of times the code must be retrieved from the bulk storage during execution.

Claims

1. A computer-implemented method for determining whether to introduce a delay in an audio stream from a particular speaker, the method comprising: Receive audio stream from sending device; providing the audio stream and the speech analysis score, information about one or more vocal emotion parameters, and one or more vocal emotion scores of a first user associated with the sending device as inputs to a trained machine learning model, wherein the trained machine learning model is iteratively applied to the audio stream, and wherein each iteration corresponds to a respective portion of the audio stream; generating, as an output, a harmfulness of the audio stream using the trained machine learning model; and The audio stream is transmitted to a receiving device, wherein the transmitting is performed to introduce a time delay in the audio stream based on the degree of harmfulness.

2. The method according to claim 1, further comprising: identifying harmful instances in the audio stream; as well as The unwanted instances in the audio stream are replaced with noise or silence before the audio stream is sent to the receiving device.

3. The method according to claim 1, further comprising: identifying silence or pauses between words in the audio stream, the silence or the pauses corresponding to specific timestamps in the audio stream; Wherein, the time delay is introduced at the specific time stamp of the silence or the pause between words as a gap in the audio stream.

4. The method according to claim 2, further comprising: Based on identifying the objectionable instance in the audio stream, the speech analysis score is updated.

5. The method according to claim 1, further comprising: receiving text from a text channel associated with the sending device, wherein the text channel is separate from the audio stream; and generating a text score indicating a level of harmfulness of the text; Wherein, the input of the trained machine learning model also includes the text score.

6. The method according to claim 1, wherein: The input to the trained machine learning model also includes the first user's harmful history, speaker history and metadata associated with the first user, and listener history and metadata associated with a second user, the second user being associated with the receiving device.

7. The method according to claim 1, wherein: The one or more vocal emotion parameters include tone, pitch, and vocal intensity level determined based on one or more previous audio streams from the transmitting device.

8. The method according to claim 1, wherein: The audio stream is provided together with a visual signal, and the method further comprises synchronizing the visual signal with the audio stream by introducing a time delay in the visual signal that is the same as the time delay of the audio stream.

9. The method according to claim 1, wherein: The audio stream is part of a video stream, and the method further comprises: analyzing the audio stream to identify harmful instances; In response to identifying the harmful instance, detecting a portion of the video stream depicting offensive behavior, wherein the offensive behavior occurs within a predetermined time period of the harmful instance; and In response to detecting the offensive behavior, at least the portion of the video stream is modified by one or more of: blurring the portion, or replacing the portion with pixels matching a background area.

10. The method according to claim 1, wherein: The audio stream is part of a video stream, and the method further comprises: performing motion detection on the video stream to detect offensive gestures; and In response to detecting the offensive behavior, at least the portion of the video stream is modified by one or more of blurring the portion or replacing the portion with pixels matching a background area.

11. The method according to claim 1, wherein: If the harmfulness level is below a minimum threshold, the time delay is zero seconds.

12. An apparatus comprising: processor; as well as A memory coupled to the processor, wherein the memory stores instructions, and when the instructions are executed by the processor, the processor performs operations, the operations comprising: Receive audio stream from sending device; providing the audio stream and the speech analysis score, information about one or more vocal emotion parameters, and one or more vocal emotion scores of a first user associated with the sending device as inputs to a trained machine learning model, wherein the trained machine learning model is iteratively applied to the audio stream, and wherein each iteration corresponds to a respective portion of the audio stream; generating, as an output, a harmfulness of the audio stream using the trained machine learning model; and The audio stream is transmitted to a receiving device, wherein the transmitting is performed to introduce a time delay in the audio stream based on the degree of harmfulness.

13. The device according to claim 12, wherein: The operations also include: identifying harmful instances in the audio stream; and The unwanted instances in the audio stream are replaced with noise or silence before the audio stream is sent to the receiving device.

14. The apparatus according to claim 12, wherein: The operations also include: identifying silence or pauses between words in the audio stream, the silence or the pauses corresponding to specific timestamps in the audio stream; Wherein, the time delay is introduced at the specific time stamp of the silence or the pause between words as a gap in the audio stream.

15. The apparatus according to claim 12, wherein: The operations also include: Based on identifying the objectionable instance in the audio stream, the speech analysis score is updated.

16. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more computers, causing the one or more computers to perform operations comprising: Receive audio stream from sending device; providing the audio stream and the speech analysis score, information about one or more vocal emotion parameters, and one or more vocal emotion scores of a first user associated with the sending device as inputs to a trained machine learning model, wherein the trained machine learning model is iteratively applied to the audio stream, and wherein each iteration corresponds to a respective portion of the audio stream; generating, as an output, a harmfulness of the audio stream using the trained machine learning model; and The audio stream is transmitted to a receiving device, wherein the transmitting is performed to introduce a time delay in the audio stream based on the degree of harmfulness.

17. The computer-readable medium of claim 16, wherein: The operations also include: identifying harmful instances in the audio stream; and The unwanted instances in the audio stream are replaced with noise or silence before the audio stream is sent to the receiving device.

18. The computer-readable medium of claim 16, wherein: The operations also include: identifying silence or pauses between words in the audio stream, the silence or the pauses corresponding to specific timestamps in the audio stream; Wherein, the time delay is introduced at the specific time stamp of the silence or the pause between words as a gap in the audio stream.

19. The computer-readable medium of claim 16, wherein: The operations also include: Based on identifying the objectionable instance in the audio stream, the speech analysis score is updated.

20. The computer-readable medium of claim 16, wherein: The operations also include: receiving text from a text channel associated with the sending device, wherein the text channel is separate from the audio stream; and generating a text score indicating a level of harmfulness of the text; Wherein, the input of the trained machine learning model also includes the text score.

Images