SYSTEM AND METHOD FOR HANDLING VOICE AUDIO STREAM INTERRUPTIONS - Patent application
Patent Information
- Application Number
- JP2023546311
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-02-03
- Filing Date
- 2021-12-09
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-12-09
AI Technical Summary
Network issues during online conferences can cause frame loss in voice and video streams, leading to irrecoverable information loss and negatively impacting user experience, as participants may miss important audio content and need to guess or ask for repetitions.
A system that includes processors configured to receive voice and text streams, generating output based on the text stream in response to audio stream interruptions, using speech-to-text conversion and text-to-speech conversion with metadata to maintain communication.
Reduces information loss during network interruptions by providing synthesized speech and text outputs, ensuring continuous participation in online meetings without missing critical information.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[Technical field]
[0001] Claiming priority This application claims the benefit of priority to commonly owned U.S. Non-Provisional Patent Application No. 17 / 166,250, filed February 3, 2021, the entire contents of which are expressly incorporated herein by reference.
[0002] TECHNICAL FIELD This disclosure relates generally to systems and methods for handling voice audio stream interruptions. [Background technology]
[0003] Advances in technology have resulted in smaller and more powerful computing devices. For example, there are now a variety of portable personal computing devices, including wireless telephones, such as mobile phones and smartphones, tablet and laptop computers, that are small, lightweight, and easily carried by users. These devices can communicate voice and data packets over wireless networks. In addition, many such devices incorporate additional features, such as digital still cameras, digital video cameras, digital recorders, and audio file players. Such devices can also process executable instructions, including software applications, such as web browser applications that may be used to access the Internet. Thus, these devices can include significant computing power.
[0004] Such computing devices often incorporate the capability to receive audio signals from one or more microphones. For example, the audio signals may represent a user voice captured by the microphone, an external sound captured by the microphone, or a combination thereof. Such devices may include communication devices used for online conferences or calls. Network problems during an online conference between a first user and a second user may cause frame loss such that some audio and video frames sent by the first device of the first user are not received by the second device of the second user. Frame loss due to network problems may lead to irrecoverable information loss during the online conference. For example, the second user must guess what he missed or ask the first user to repeat what he missed, which negatively impacts the user experience. Summary of the Invention [Means for solving the problem]
[0005] According to one implementation of the present disclosure, a device for communication includes one or more processors configured to receive a voice audio stream representing a voice of a first user during an online conference. The one or more processors are also configured to receive a text stream representing the voice of the first user. The one or more processors are further configured to selectively generate an output based on the text stream in response to an interruption of the voice audio stream.
[0006] According to another implementation of the present disclosure, a method of communication includes receiving, at a device, a voice audio stream representing a voice of a first user during an online conference. The method also includes receiving, at the device, a text stream representing the voice of the first user. The method further includes selectively generating, at the device, an output based on the text stream in response to an interruption of the voice audio stream.
[0007] According to another implementation of the present disclosure, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to receive a voice audio stream representing a voice of a first user during an online conference. The instructions, when executed by the one or more processors, also cause the one or more processors to receive a text stream representing the voice of the first user. The instructions, when executed by the one or more processors, further cause the one or more processors to selectively generate an output based on the text stream in response to an interruption of the voice audio stream.
[0008] According to another implementation of the present disclosure, an apparatus includes means for receiving a voice audio stream during an online conference, the voice audio stream representing a speech of a first user. The apparatus also includes means for receiving a text stream representing the speech of the first user. The apparatus further includes means for selectively generating an output based on the text stream in response to an interruption of the voice audio stream.
[0009] Other aspects, advantages, and features of the present disclosure will become apparent after consideration of the entire disclosure, including the following sections: Brief Description of the Drawings, Detailed Description, and the Claims. [Brief description of the drawings]
[0010] [Figure 1]1 is a block diagram of a particular illustrative aspect of a system operable to handle voice audio stream interruptions, in accordance with certain examples of the present disclosure. [Diagram 2] 1 is a diagram of an example aspect of a system operable to handle voice audio stream interruptions, in accordance with some examples of the present disclosure. [Figure 3A] 3 is a diagram of an example graphical user interface (GUI) generated by the system of FIG. 1 or the system of FIG. 2, in accordance with some examples of the present disclosure. [Figure 3B] 3 is a diagram of an example GUI generated by the system of FIG. 1 or the system of FIG. 2 in accordance with some examples of the present disclosure. [Figure 3C] 3 is a diagram of an example GUI generated by the system of FIG. 1 or the system of FIG. 2 in accordance with some examples of the present disclosure. [Figure 4A] 3 is a diagram of an exemplary aspect of the operation of the system of FIG. 1 or the system of FIG. 2 in accordance with some examples of the present disclosure. [Figure 4B] 3 is a diagram of an exemplary aspect of the operation of the system of FIG. 1 or the system of FIG. 2 in accordance with some examples of the present disclosure. [Diagram 5] 1 is a diagram of an example aspect of a system operable to handle voice audio stream interruptions, in accordance with some examples of the present disclosure. [Figure 6A] 6 is a diagram of an example graphical user interface (GUI) generated by the system of FIG. 5, in accordance with some examples of the present disclosure. [Figure 6B] 6 is a diagram of an example GUI generated by the system of FIG. 5 in accordance with some examples of the present disclosure. [Figure 6C] 6 is a diagram of an example GUI generated by the system of FIG. 5 in accordance with some examples of the present disclosure. [Figure 7A] 6 is a diagram of an example aspect of the operation of the system of FIG. 5 in accordance with some examples of the present disclosure. [Figure 7B] 6 is a diagram of an example aspect of the operation of the system of FIG. 5 in accordance with some examples of the present disclosure. [Figure 8] FIG. 6 is a diagram of a particular implementation of a method for handling voice audio stream interruptions that may be performed by any of the systems of FIG. 1, FIG. 2, or FIG. 5, in accordance with some examples of the present disclosure. [Figure 9] FIG. 2 illustrates an example of an integrated circuit operable to handle voice audio stream interruptions, in accordance with some examples of the present disclosure. [Figure 10] FIG. 1 is a diagram of a mobile device operable to handle voice audio stream interruptions, according to some examples of the present disclosure. [Figure 11] FIG. 1 is a diagram of a headset operable to handle voice audio stream interruptions in accordance with some examples of the present disclosure. [Figure 12] FIG. 1 illustrates a diagram of a wearable electronic device operable to handle voice audio stream interruptions, according to some examples of the present disclosure. [Figure 13] FIG. 1 illustrates a diagram of a voice-controlled speaker system operable to handle voice audio stream interruptions, according to some examples of the present disclosure. [Figure 14] 1 is a diagram of a camera operable to handle voice audio stream interruptions, according to some examples of the present disclosure. [Figure 15] FIG. 1 is a diagram of a headset, such as a virtual reality headset or an augmented reality headset, operable to handle voice audio stream interruptions, in accordance with some examples of the present disclosure. [Figure 16] 1 is a diagram of a first example of a vehicle operable to handle voice audio stream interruptions, according to some examples of the present disclosure. [Figure 17] FIG. 11 is a diagram of a second example of a vehicle operable to handle a voice audio stream interruption, according to some examples of the present disclosure. [Figure 18] 1 is a block diagram of a particular illustrative example of a device operable to handle voice audio stream interruptions, in accordance with certain examples of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Missing a portion of an online conference or call can negatively impact a user experience. For example, during an online conference between a first user and a second user, if some audio frames sent by a first device of the first user are not received by a second device of the second user, the second user may miss a portion of the first user's voice. The second user must guess what the first user said or ask the first user to repeat what he or she missed. This can cause misunderstandings, disrupt the flow of the conversation, and waste time.
[0012] A system and method for handling voice audio stream interruptions is disclosed. For example, each device includes a conference manager configured to establish an online conference or call between the device and one or more other devices. The interruption manager (in the device or in the server) is configured to handle the voice audio stream interruptions.
[0013] During an online conference between a first device of a first user and a second device of a second user, a conference manager of the first device sends a media stream to the second device. The media stream includes a voice audio stream, a video stream, or both. The voice audio stream corresponds to the voice of the first user during the conference.
[0014] A stream manager (at the first device or at the server) generates a text stream by performing speech-to-text conversion on the voice audio stream and forwards the text stream to the second device. The stream manager (e.g., a conference manager at the first device or at the server) forwards the text stream simultaneously with the media stream throughout the online conference in a first operating mode (e.g., a caption data transmission mode). In an alternative example, the stream manager (e.g., an interruption manager at the first device or at the server) forwards the text stream to the second device in response to detecting a network problem (e.g., low bandwidth, packet loss, etc.) simultaneously with sending the media stream to the second device in a second operating mode (e.g., an interruption data transmission mode).
[0015] In some examples, the network problem causes an interruption in reception of the media stream at the second device without an interruption in reception of the text stream. In some examples, the second device provides the text stream to a display in a first operating mode (e.g., a caption data display mode) independent of detecting a network problem. In other examples, the second device displays the text stream in response to detecting an interruption in the media stream in a second operating mode (e.g., an interrupted data display mode).
[0016] In a particular example, a stream manager (e.g., a conference manager or an interruption manager) transfers a metadata stream in addition to the text data. The metadata indicates emotion, intonation, and other attributes of the first user's voice. In a particular example, the second device displays the metadata stream in addition to the text stream. For example, the text stream is annotated based on the metadata stream.
[0017] In certain examples, the second device performs text-to-speech conversion on the text stream to generate a synthetic speech audio stream and outputs the synthetic speech audio stream (e.g., to replace the interrupted speech audio stream). In certain examples, the text-to-speech conversion is based at least in part on the metadata stream.
[0018] In a particular example, the second device displays an avatar (e.g., to replace an interrupted video stream) during output of the synthetic voice audio stream. In a particular example, the text-to-speech conversion is based on a generic voice model. For example, a first generic voice model may be used for one user and a second generic voice model may be used for another user so that a listener can distinguish voices corresponding to different users. In another particular example, the text-to-speech conversion is based on a user voice model generated based on the voice of the first user. In a particular example, the user voice model is generated prior to the online conference. In a particular example, the user voice model is generated (or updated) during the online conference. In a particular example, the user voice model is initialized from the generic voice model and updated based on the voice of the first user.
[0019] In a particular example, the avatar indicates that a voice model is being trained. For example, the avatar is initialized as red, which indicates that a generic voice model is being used (or that a user voice model is not ready), and over time the avatar transitions from red to green, which indicates that a voice model is being trained. The green avatar indicates that a user voice model has been trained (or that a user voice model is ready).
[0020] An online conference can be between two or more users. In a situation where a first device is experiencing network problems but a third device of a third user in the online conference is not experiencing network problems, the second device can output a synthesized voice audio stream for the first user simultaneously with outputting a second media stream received from the third device corresponding to the third user's voice, video, or both.
[0021] Certain aspects of the present disclosure are described below with reference to the drawings. In this description, common features are designated by common reference numbers. Various terms used herein are used only for the purpose of describing particular implementations and are not intended to limit the implementations. For example, the singular forms "a," "an," and "the" are intended to include the plural, unless the context clearly indicates otherwise. Furthermore, some features described herein are singular in some implementations and plural in other implementations. To illustrate, FIG. 1 illustrates a device 104 including one or more processors ("processor(s)" 160 in FIG. 1), indicating that in some implementations the device 104 includes a single processor 160 and in other implementations the device 104 includes multiple processors 160.
[0022] As used herein, the terms "comprise," "comprises," and "comprising" may be used interchangeably with "include," "includes," or "including." Additionally, the term "wherein" may be used interchangeably with "where." As used herein, "exemplary" indicates an example, an implementation, and / or an aspect, and should not be construed as limiting or indicating a preferred or preferred implementation. As used herein, orthogonal terms (e.g., "first," "second," "third," etc.) used to modify an element, such as a structure, component, operation, etc., do not in themselves indicate any priority or order of the element relative to another element, but rather merely distinguish the element from another element having the same name (apart from the use of orthogonal terms). As used herein, the term "set" refers to one or more of a particular element, and the term "plurality" refers to a plurality (e.g., two or more) of a particular element.
[0023] As used herein, "coupled" may include "communicatively coupled," "electrically coupled," or "physically coupled," as well as (or alternatively) any combination thereof. Two devices (or components) may be directly or indirectly coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) via one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks, or combinations thereof), and the like. Two devices (or components) that are electrically coupled may be included in the same device or in different devices, and may be connected via electronic circuits, one or more connectors, or inductive coupling, as illustrative and non-limiting examples. In some implementations, two devices (or components) that are communicatively coupled, such as in electrical communication, may transmit and receive signals (e.g., digital or analog signals) directly or indirectly via one or more wires, buses, networks, and the like. As used herein, "directly coupled" may include two devices that are coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) with no intervening components.
[0024] In this disclosure, terms such as "determine," "calculate," "estimate," "shift," "adjust," and the like may be used to describe how one or more operations are performed. It should be noted that such terms should not be construed as limiting, and other techniques may be utilized to perform similar operations. In addition, "generate," "calculate," "estimate," "use," "select," "access," and "determine" referred to herein may be used interchangeably. For example, "generating," "calculating," "estimating," or "determining" a parameter (or signal) may refer to actively generating, estimating, calculating, or determining a parameter (or signal), or may refer to using, selecting, or accessing a parameter (or signal) that has already been generated by another component or device, etc.
[0025] 1, a particular example embodiment of a system configured to handle voice audio stream interruptions is disclosed, generally designated 100. System 100 includes a device 102 coupled to a device 104 via a network 106. Network 106 includes a wired network, a wireless network, or both. Device 102 is coupled to a camera 150, a microphone 152, or both. Device 104 is coupled to a speaker 154, a display device 156, or both.
[0026] The device 104 includes one or more processors 160 coupled to the memory 132. The one or more processors 160 include a conference manager 162 coupled to an interruption manager 164. The conference manager 162 and the interruption manager 164 are coupled to a graphical user interface (GUI) generator 168. The interruption manager 164 includes a text-to-speech converter 166. The device 102 includes one or more processors 120 including a conference manager 122 coupled to an interruption manager 124. The conference manager 122 and the conference manager 162 are configured to establish an online conference (e.g., an audio call, a video call, a conference call, etc.). In a particular example, the conference manager 122 and the conference manager 162 correspond to a client of a communication application (e.g., an online conference application). The interruption manager 124 and the interruption manager 164 are configured to process voice audio interruptions.
[0027] In some implementations, conference manager 122 and conference manager 162 are blind to (e.g., unaware of) any voice audio interruptions managed by interrupt manager 124 and interrupt manager 164. In some implementations, conference manager 122 and conference manager 162 correspond to an upper layer (e.g., application layer) of a network protocol stack (e.g., Open Systems Interconnection (OSI) model) of device 102 and device 104, respectively. In some implementations, interrupt manager 124 and interrupt manager 164 correspond to a lower level (e.g., transport layer) of a network protocol stack of device 102 and device 104, respectively.
[0028] In some implementations, the device 102, the device 104, or both, correspond to or are included in various types of devices. In an illustrative example, the one or more processors 120, the one or more processors 160, or a combination thereof are integrated into a headset device, such as that further described with reference to FIG. 11. In other examples, the one or more processors 120, the one or more processors 160, or a combination thereof are integrated into at least one of a mobile phone or tablet computing device, such as that described with reference to FIG. 10, a wearable electronic device, such as that described with reference to FIG. 12, a voice-controlled speaker system, such as that described with reference to FIG. 13, a camera device, such as that described with reference to FIG. 14, or a virtual reality headset, an augmented reality headset, or a mixed reality headset, such as that described with reference to FIG. 15. In another illustrative example, the one or more processors 120, the one or more processors 160, or a combination thereof are integrated into a vehicle, such as that further described with reference to FIG. 16 and FIG. 17.
[0029] During operation, the conference manager 122 and the conference manager 162 establish an online conference (e.g., an audio call, a video call, a conference call, or a combination thereof) between the device 102 and the device 104. For example, the online conference is between the user 142 of the device 102 and the user 144 of the device 104. The microphone 152 captures the voice of the user 142 while the user 142 is speaking and provides an audio input 153 representative of the voice to the device 102. In certain aspects, the camera 150 (e.g., a still camera, a video camera, or both) captures one or more images (e.g., still images or videos) of the user 142 and provides a video input 151 representative of the one or more images to the device 102. In certain aspects, the camera 150 provides the video input 151 to the device 102 at the same time that the microphone 152 provides the audio input 153 to the device 102.
[0030] Conference manager 122 generates media stream 109 of media frames based on audio input 153, video input 151, or both. For example, media stream 109 includes voice audio stream 111, video stream 113, or both. In certain aspects, conference manager 122 sends media stream 109 in real time to device 104 over network 106. For example, conference manager 122 generates media frames of media stream 109 as video input 151, audio input 153, or both are being received, and sends (e.g., begins transmitting) media stream 109 of media frames once the media frames are generated.
[0031] In certain implementations, conference manager 122 generates text stream 121, metadata stream 123, or both based on audio input 153 during a first operational mode (e.g., caption data transmission mode) of device 102. For example, conference manager 122 performs speech-to-text conversion on audio input 153 to generate text stream 121. Text stream 121 indicates text corresponding to speech detected in audio input 153. In certain aspects, conference manager 122 performs speech intonation analysis on audio input 153 to generate metadata stream 123. For example, metadata stream 123 indicates speech intonation (e.g., emotion, pitch, tone, or combinations thereof) detected in audio input 153. In a first operating mode of device 102 (e.g., a caption data transmission mode), conference manager 122 sends text stream 121, metadata stream 123, or both (e.g., as subtitling data) along with media stream 109 to device 104 (e.g., regardless of network problems or voice audio interruptions). Alternatively, conference manager 122 refrains from generating text stream 121 and metadata stream 123 in response to determining that no voice audio interruption is detected during a second operating mode of device 102 (e.g., a interrupted data transmission mode).
[0032] The device 104 receives a media stream 109 of media frames from the device 102 over the network 106. In a particular implementation, the device 104 receives a set (e.g., a burst) of media frames of the media stream 109. In an alternative implementation, the device 104 receives one media frame of the media stream 109 at a time. The conference manager 162 plays out the media frames of the media stream 109. For example, the conference manager 162 generates an audio output 143 based on the voice audio stream 111 and plays out the audio output 143 through the speaker 154 (e.g., as streaming audio content). In a particular aspect, the GUI generator 168 generates a GUI 145 based on the media stream 109, as further described with reference to FIG. 3A. For example, the GUI generator 168 generates (or updates) the GUI 145 to display video content of the video stream 113 and provides the GUI 145 to the display device 156 (e.g., streaming video content). User 144 can view an image of user 142 on display device 156 while hearing the audio of user 142 through speaker 154 .
[0033] In certain implementations, the conference manager 162 stores media frames of the media stream 109 in a buffer prior to playout. For example, the conference manager 162 adds a delay between receiving a media frame and playing the media frame at a first play time to increase the likelihood that a subsequent media frame is available in the buffer at the corresponding play time (e.g., the second play time). In certain aspects, the conference manager 162 plays out the media stream 109 in real time. For example, the conference manager 162 retrieves media frames of the media stream 109 from the buffer to play out the audio output 143, the video content of the GUI 145, or both, while a subsequent media frame of the media stream 109 is being received (or is expected to be received) by the device 104.
[0034] The conference manager 162 plays out the text stream 121 along with the media stream 109 (e.g., independent of detecting an interruption in the audio audio stream 111) during a first operating mode (e.g., a caption data display mode) of the device 104. In a particular aspect, the conference manager 162 receives the text stream 121, the metadata stream 123, or both along with the media stream 109, for example, during a first operating mode (e.g., a caption data transmission mode) of the device 102. In an alternative aspect, the conference manager 162 does not receive the text stream 121, the metadata stream 123, or both, and generates the text stream 121, the metadata stream 123, or both based on the audio audio stream 111, the video stream 113, or both, for example, during a second operating mode (e.g., an interrupted data transmission mode) of the device 102. For example, the conference manager 162 performs speech-to-text conversion on the speech audio stream 111 to generate the text stream 121 and performs intonation analysis on the speech audio stream 111 to generate the metadata stream 123 .
[0035] During a first operational mode (e.g., a caption data display mode) of device 104, conference manager 162 provides text stream 121 as an output to display device 156. For example, conference manager 162 displays the text content of text stream 121 (e.g., as subtitles) using GUI 145 simultaneously with displaying the video content of video stream 113, providing audio output 143 to speaker 154, or both. Illustratively, conference manager 162 provides text stream 121 to GUI generator 168 simultaneously with providing video stream 113 to GUI generator 168. GUI generator 168 updates GUI 145 to display text stream 121, video stream 113, or both. GUI generator 168 provides updates of GUI 145 to display device 156 simultaneously with conference manager 162 providing voice audio stream 111 as audio output 143 to speaker 154.
[0036] In a particular example, conference manager 162 generates annotated text stream 137 based on text stream 121 and metadata stream 123. In a particular aspect, conference manager 162 generates annotated text stream 137 by adding annotations to text stream 121 based on metadata stream 123. Conference manager 162 provides annotated text stream 137 as an output to display device 156. For example, conference manager 162 plays out annotated text stream 137 along with media stream 109. To illustrate, conference manager 162 displays the annotated text content of annotated text stream 137 using GUI 145 (e.g., as subtitles with intonation indication) simultaneously with displaying video content in video stream 113, providing audio output 143 to speaker 154, or both.
[0037] In certain implementations, conference manager 162 refrains from playing out text stream 121 (e.g., annotated text stream 137) in the second operating mode (e.g., interrupted data display mode or subtitle disabled mode) of device 104. For example, conference manager 162 does not receive text stream 121 (e.g., during the second operating mode of device 102) and does not generate text stream 121 in the second operating mode (e.g., interrupted data display mode or subtitle disabled mode). As another example, conference manager 162 refrains from playing out text stream 121 (e.g., annotated text stream 137) in response to receiving text stream 121 and detecting the second operating mode (e.g., interrupted data display mode or subtitle disabled mode) of device 104. In certain aspects, in a second operating mode (e.g., an interrupted data display mode) of the device 104, the interruption manager 164 refrains from playing out the text stream 121 (e.g., the annotated text stream 137) in response to determining that no interruption has been detected in the media stream 109 (e.g., a portion of the media stream 109 corresponding to the text stream 121 has been received).
[0038] In certain aspects, the interruption manager 164 initializes the voice model 131, such as an artificial neural network, based on a generic voice model prior to or near the start of the online conference. In certain aspects, the interruption manager 164 selects a generic voice model from a plurality of generic voice models based on a determination that the generic voice model matches (e.g., is associated with) demographic data of the user 142, such as the user's age, location, gender, or a combination thereof. In certain aspects, the interruption manager 164 predicts demographic data based on the contact information (e.g., name, location, phone number, address, or a combination thereof) of the user 142 prior to the online conference (e.g., a scheduled conference). In certain aspects, the interruption manager 164 estimates demographic data based on the voice audio stream 111, the video stream 113, or both, during the beginning portion of the online conference. For example, the interruption manager 164 analyzes the voice audio stream 111, the video stream 113, or both, to estimate the user 142's age, regional accent, gender, or a combination thereof. In certain aspects, the interruption manager 164 retrieves a (eg, previously generated) speech model 131 associated with the user 142 (eg, matching a user identifier of the user 142).
[0039] In a particular aspect, the interruption manager 164 trains (e.g., generates or updates) the speech model 131 based on the speech detected in the voice audio stream 111 during the online conference (e.g., prior to the interruption of the voice audio stream 111). Illustratively, the text-to-speech converter 166 is configured to use the speech model 131 to perform text-to-speech conversion. In a particular aspect, the interruption manager 164 receives (e.g., during a first mode of operation of the device 102) or generates (e.g., during a second mode of operation of the device 102) the text stream 121, the metadata stream 123, or both, corresponding to the voice audio stream 111. The text-to-speech converter 166 uses the speech model 131 to generate the synthetic voice audio stream 133 by performing text-to-speech conversion on the text stream 121, the metadata stream 123, or both. The interruption manager 164 uses a training technique to update the speech model 131 based on a comparison of the voice audio stream 111 and the synthetic voice audio stream 133. In an illustrative example in which the speech model 131 includes an artificial neural network, the interruption manager 164 uses backpropagation to update the weights and biases of the speech model 131. According to some aspects, the speech model 131 is updated such that subsequent text-to-speech conversions using the speech model 131 are more likely to generate synthetic speech that closely matches the voice characteristics of the user 142.
[0040] In certain aspects, the interruption manager 164 generates an avatar 135 (e.g., a visual representation) of the user 142. In certain aspects, the avatar 135 includes or corresponds to a training indicator that indicates a level of training of the voice model 131, as further described with reference to FIGS. 3A-3C. For example, in response to a determination that a first training criterion is not met, the interruption manager 164 initializes the avatar 135 to a first visual representation indicating that the voice model 131 is not trained. During the online conference, in response to a determination that the first training criterion is met and a second training criterion is not met, the interruption manager 164 updates the avatar 135 from the first visual representation to a second visual representation indicating that training of the voice model 131 is in progress. In response to a determination that the second training criterion is met, the interruption manager 164 updates the avatar 135 to a third visual representation indicating that training of the voice model 131 is complete.
[0041] The training criteria may be based on a count of audio samples used to train the voice model 131, a playback duration of the audio samples used to train the voice model 131, a coverage of the audio samples used to train the voice model 131, a success metric of the voice model 131, or a combination thereof. In certain aspects, the coverage of the audio samples used to train the voice model 131 corresponds to the distinct sounds (e.g., vowels, consonants, etc.) represented by the audio samples. In certain aspects, the success metric is based on a comparison of the audio samples used to train the voice model 131 and a synthetic speech generated based on the voice model 131 (e.g., a match between the audio samples and the synthetic speech).
[0042] According to some implementations, a first color, a first shading, a first size, a first animation, or a combination thereof, of the avatar 135 indicates that the voice model 131 is not trained. A second color, a second shading, a second size, a second animation, or a combination thereof, of the avatar 135 indicates that the voice model 131 is partially trained. A third color, a third shading, a third size, a third animation, or a combination thereof, of the avatar 135 indicates that training of the voice model 131 is complete. In certain aspects, the GUI generator 168 generates (or updates) the GUI 145 to show a visual representation of the avatar 135.
[0043] In certain aspects, the interruption manager 124 detects a network problem (e.g., reduced bandwidth) in a communication link to the device 104. In response to detecting the network problem, the interruption manager 124 sends an interruption notification 119 to the device 104 indicating the interruption of the voice audio stream 111, refrains from sending (e.g., stops sending) subsequent media frames of the media stream 109 to the device 104 until it detects that the network problem has been resolved, or both. For example, in response to detecting the network problem, the interruption manager 124 refrains from sending (e.g., stops sending) the voice audio stream 111, the video stream 113, or both to the device 104 until the end of the interruption.
[0044] The interruption manager 124 sends the text stream 121, the metadata stream 123, or both corresponding to the subsequent media frames. For example, the interruption manager 124 continues to send the text stream 121, the metadata stream 123, or both corresponding to the subsequent media frames in a first operating mode (e.g., caption data transmission mode) of the device 102. Illustratively, in the first operating mode (e.g., caption data transmission mode), the conference manager 122 generates the media stream 109, the text stream 121, the metadata stream 123, or a combination thereof. In response to detecting a network problem in the first operating mode (e.g., caption data transmission mode), the interruption manager 124 stops sending subsequent media frames of the media stream 109 and continues sending the text stream 121, the metadata stream 123, or both corresponding to the subsequent media frames to the device 104. Alternatively, in response to detecting a network problem in the second operating mode (e.g., the interrupted data transmission mode) of the device 102, the interrupt manager 124 generates the text stream 121, the metadata stream 123, or both, corresponding to the subsequent media frames based on the audio input 153. Illustratively, in the second operating mode (e.g., the interrupted data transmission mode), the conference manager 122 generates the media stream 109 and does not generate the text stream 121, the metadata stream 123, or both. In response to detecting a network problem in the second operating mode (e.g., the interrupted data transmission mode) of the device 102, the interrupt manager 124 stops transmitting the subsequent media frames of the media stream 109 and starts transmitting the text stream 121, the metadata stream 123, or both, corresponding to the subsequent media frames to the device 104. In certain aspects, in a second operating mode of device 102 (e.g., an interrupted data transmission mode), sending text stream 121, metadata stream 123, or both to device 104 corresponds to sending interruption notification 119 to device 104.
[0045] In certain aspects, the interruption manager 164 detects an interruption in the audio audio stream 111 in response to receiving an interruption notification 119 from the device 102. In certain aspects, when the device 102 is operating in the second operating mode (e.g., the interrupted data transmission mode), the interruption manager 164 detects an interruption in the audio audio stream 111 in response to receiving the text stream 121, the metadata stream 123, or both.
[0046] In certain aspects, the interruption manager 164 detects an interruption in the voice audio stream 111 in response to determining that an audio frame of the voice audio stream 111 was not received within a threshold duration of a last received audio frame of the voice audio stream 111. For example, the last received audio frame of the voice audio stream 111 is received at the device 104 at a first reception time. The interruption manager 164 detects the interruption in response to determining that an audio frame of the voice audio stream 111 was not received within a threshold duration of the first reception time. In certain aspects, the interruption manager 164 sends an interruption notification to the device 102. In certain aspects, the interruption manager 124 detects a network problem in response to receiving an interruption notification from the device 104. In response to detecting the network problem, the interruption manager 124 sends the text stream 121, the metadata stream 123, or both, to the device 104 (e.g., instead of sending a subsequent media frame of the media stream 109), as described above.
[0047] The interruption manager 164 selectively generates an output based on the text stream 121 in response to detecting the interruption. For example, the interruption manager 164 provides the text stream 121, the metadata stream 123, the annotated text stream 137, or a combination thereof to the text-to-speech converter 166 in response to the interruption. The text-to-speech converter 166 generates the synthetic speech audio stream 133 by using the speech model 131 to perform text-to-speech conversion based on the text stream 121, the metadata stream 123, the annotated text stream 137, or a combination thereof. For example, the synthetic speech audio stream 133 based on the text stream 121 and independent of the metadata stream 123 corresponds to the speech indicated by the text stream 121 with the neural speech characteristics of the user 142 as represented by the speech model 131. As another example, the synthetic speech audio stream 133 based on the annotated text stream 137 (e.g., text stream 121 and metadata stream 123) corresponds to the voice indicated by the text stream 121 with the voice characteristics of the user 142, as represented by the speech model 131 with the intonation indicated by the metadata stream 123. Using the speech model 131, at least partially trained on the voice of the user 142 (e.g., the speech audio stream 111) to perform the text-to-speech conversion, allows the synthetic speech audio stream 133 to better match the voice characteristics of the user 142. In response to the interruption, the interruption manager 164 provides the synthetic speech audio stream 133 as an audio output 143 to the speaker 154, stops playing the speech audio stream 111, stops playing the video stream 113, or a combination thereof.
[0048] In certain aspects, the interruption manager 164 selectively displays the avatar 135 while simultaneously providing the synthetic voice audio stream 133 to the speaker 154 as the audio output 143. For example, the interruption manager 164 refrains from displaying the avatar 135 while providing the voice audio stream 111 to the speaker 154 as the audio output 143. As another example, the interruption manager 164 displays the avatar 135 while providing the synthetic voice audio stream 133 to the speaker 154 as the audio output 143. Illustratively, the GUI generator 168 updates the GUI 145 to display the avatar 135 instead of the video stream 113 while the synthetic voice audio stream 133 is output as the audio output 143 for playout by the speaker 154. In certain aspects, the interruption manager 164 displays a first representation of the avatar 135 simultaneously with providing the voice audio stream 111 as audio output 143 to the speaker 154 and displays a second representation of the avatar 135 simultaneously with providing the synthetic voice audio stream 133 as audio output 143 to the speaker 154. For example, as further described with reference to FIG. 3C , the first representation indicates that the avatar 135 is being trained or has been trained (e.g., a training indicator for the voice model 131) and the second representation indicates that the avatar 135 is speaking (e.g., the voice model 131 is being used to generate synthetic speech).
[0049] In certain implementations, the interruption manager 164 selectively provides the text stream 121, the annotated text stream 137, or both as output to the display device 156. For example, the interruption manager 164 provides the text stream 121, the annotated text stream 137, or both to the GUI generator 168 for updating the GUI 145 to display the text stream 121, the annotated text stream 137, or both, in response to an interruption during the second operating mode (e.g., the interrupted data display mode) of the device 104. In alternative implementations, the interruption manager 164 continues to provide the text stream 121, the annotated text stream 137, or both as output to the display device 156 (e.g., independent of an interruption) during the first operating mode (e.g., the caption data display mode) of the device 104. In certain aspects, the interruption manager 164 provides the text stream 121 , the annotated text stream 137 , or both, to the display device 156 while simultaneously providing the synthesized speech audio stream 133 as audio output 143 to the speaker 154 .
[0050] In certain implementations, the interruption manager 164 outputs one or more of the synthetic speech audio stream 133, the text stream 121, or the annotated text stream 137 based on the interruption configuration setting and in response to the interruption. For example, the interruption manager 164 provides the synthetic speech audio stream 133 as audio output 143 to the speaker 154 while simultaneously providing the text stream 121, the annotated text stream 137, or both, to the display device 156 in response to the interruption and in response to determining that the interruption configuration setting has a first value (e.g., 0 or “audio and text”). The interruption manager 164 provides the text stream 121, the annotated text stream 137, or both, to the display device 156 and refrains from providing the audio output 143 to the speaker 154 in response to the interruption and in response to determining that the interruption configuration setting has a second value (e.g., 1 or “text only”). In response to the interruption and in response to determining that the interruption configuration setting has a third value (e.g., 2 or “audio only”), the interruption manager 164 refrains from providing the text stream 121, the annotated text stream 137, or both, to the display device 156 and provides the synthesized speech audio stream 133 as audio output 143 to the speaker 154. In certain aspects, the interruption configuration setting is based on default data, user input, or both.
[0051] In certain aspects, the interrupt manager 124 detects that the interrupt has ended and sends an interruption end notification to the device 104. For example, the interrupt manager 124 detects that the interrupt has ended in response to determining that the available communications bandwidth of the communications link with the device 104 is greater than a threshold. In certain aspects, the interrupt manager 164 detects that the interrupt has ended in response to receiving an interruption end notification from the device 102.
[0052] In another particular aspect, the interrupt manager 164 detects that the interrupt has ended and sends an interruption end notification to the device 102. For example, the interrupt manager 164 detects that the interrupt has ended in response to determining that the available communications bandwidth of the communications link with the device 102 is greater than a threshold. In a particular aspect, the interrupt manager 124 detects that the interrupt has ended in response to receiving an interruption end notification from the device 104.
[0053] In response to detecting that the interruption has ended, the conference manager 122 resumes transmitting the voice audio stream 111, the video stream 113, or both, to the device 104. In certain aspects, the transmission of the voice audio stream 111, the video stream 113, or both, corresponds to transmitting an interruption end notification. In response to detecting that the interruption has ended during a second operating mode (e.g., an interrupted data transmission mode) of the device 102, the interruption manager 124 refrains from sending the text stream 121, the metadata stream 123, or both, to the device 104.
[0054] In response to detecting that the interruption has ended, conference manager 162 refrains from generating synthetic speech audio stream 133 based on text stream 121, refrains from providing (e.g., stops) synthetic speech audio stream 133 as audio output 143 to speaker 154, and resumes playing (e.g., providing speech audio stream 111) speech audio stream 111 as audio output 143 to speaker 154. In response to detecting that the interruption has ended, conference manager 162 resumes providing video stream 113 to display device 156. For example, conference manager 162 provides video stream 113 to GUI generator 168 for updating GUI 145 to display video stream 113.
[0055] In a particular aspect, in response to detecting that the interruption has ended, the interruption manager 164 sends a first request to the GUI generator 168 to update the GUI 145 to indicate that the voice model 131 is not being used to output synthetic voice audio (e.g., the avatar 135 is not speaking). In response to receiving the first request, the GUI generator 168 updates the GUI 145 to display a first representation of the avatar 135 indicating that the voice model 131 is being trained or has been trained and that the voice model 131 is not being used to output synthetic voice audio (e.g., the avatar 135 is not speaking). In an alternative aspect, in response to detecting that the interruption has ended, the interruption manager 164 sends a second request to the GUI generator 168 to stop displaying the avatar 135. For example, in response to receiving the second request, the GUI generator 168 updates the GUI 145 to refrain from displaying the avatar 135.
[0056] In certain aspects, in response to detecting that the interruption has ended during the second operating mode (e.g., the interrupted data display mode or the no captioned data mode), the interruption manager 164 refrains from providing the text stream 121, the annotated text stream 137, or both, to the display device 156. For example, the GUI generator 168 updates the GUI 145 to refrain from displaying the text stream 121, the annotated text stream 137, or both.
[0057] In this manner, system 100 reduces (e.g., eliminates) information loss during interruptions in voice audio stream 111 during an online conference. For example, if a network problem prevents voice audio stream 111 from being received by device 104 but text can be received by device 104, user 144 continues to receive audio corresponding to the voice of user 142 (e.g., synthesized voice audio stream 133), text (e.g., text stream 121, annotated text stream 137, or both), or a combination thereof.
[0058] Although the camera 150 and microphone 152 are shown as coupled to the device 102, in other implementations the camera 150, the microphone 152, or both may be integrated into the device 102. Although the speaker 154 and the display device 156 are shown as coupled to the device 104, in other implementations the speaker 154, the display device 156, or both may be integrated into the device 104. Although one microphone and one speaker are shown, other implementations may include one or more additional microphones configured to capture user voice, one or more additional speakers configured to output voice audio, or a combination thereof.
[0059] It should be understood that for ease of explanation, device 102 is described as a transmitting device and device 104 is described as a receiving device. During a call, the roles of device 102 and device 104 can switch when user 144 begins speaking. For example, device 104 may be the transmitting device and device 102 may be the receiving device. To illustrate, device 104 can include a microphone and a camera for capturing audio and video of user 144, and device 102 can include or be coupled to a speaker and a display for playing out audio and video to user 142. In certain aspects, for example, when both user 142 and user 144 are speaking at the same time or at overlapping times, each of device 102 and device 104 can be a transmitting device and a receiving device.
[0060] In certain aspects, conference manager 122 is also configured to perform one or more operations described with reference to conference manager 162, and vice versa. In certain aspects, interruption manager 124 is also configured to perform one or more operations described with reference to interruption manager 164, and vice versa. Although GUI generator 168 has been described as separate from conference manager 162 and interruption manager 164, in other implementations GUI generator 168 is integrated into conference manager 162, interruption manager 164, or both. To illustrate, in some examples, conference manager 162, interruption manager 164, or both are configured to perform some of the operations described with reference to GUI generator 168.
[0061] 2, a system operable to handle voice audio stream interruptions is shown and generally designated 200. In certain aspects, the system 100 of FIG.
[0062] The system 200 includes a server 204 coupled to the device 102 and the device 104 via the network 106. The server 204 includes a conference manager 122 and an interruption manager 124. The server 204 is configured to transfer online conference data from the device 102 to the device 104 and vice versa. For example, the conference manager 122 is configured to establish an online conference between the device 102 and the device 104.
[0063] The device 102 includes a conference manager 222. During an online conference, the conference manager 222 sends the media stream 109 (e.g., the voice audio stream 111, the video stream 113, or both) to the server 204. The conference manager 122 of the server 204 receives the media stream 109 (e.g., the voice audio stream 111, the video stream 113, or both) from the device 102. In a particular implementation, the device 102 sends the text stream 121, the metadata stream 123, or both at the same time as sending the media stream 109 to the server 204.
[0064] In certain aspects, subsequent operations are performed as described with reference to FIG. 1, with server 204 replacing device 102. For example, conference manager 122 (operating in server 204 instead of in device 102 as in FIG. 1) sends media stream 109, text stream 121, metadata stream 123, or a combination thereof to device 104 in a manner similar to that described with reference to FIG. 1. For example, conference manager 122 sends text stream 121, metadata stream 123, or both during a first operating mode (e.g., captioned data transmission mode) of server 204. In certain implementations, conference manager 122 forwards text stream 121, metadata stream 123, or both received from device 102 to device 104. In some implementations, conference manager 122 generates metadata stream 123 based on text stream 121, media stream 109, or a combination thereof. In these implementations, the conference manager 122 forwards the text stream 121 received from the device 102 to the device 104, sends the metadata stream 123 generated at the server 204 to the device 104, or both. In some implementations, the conference manager 122 generates the text stream 121, the metadata stream 123, or both based on the media stream 109 and forwards the text stream 121, the metadata stream 123, or both to the device 104. Alternatively, the conference manager 122 refrains from sending the text stream 121, the metadata stream 123, or both in response to determining that no interruption is detected during a second operating mode (e.g., an interrupted data transmission mode) of the server 204. The device 104 receives the media stream 109, the text stream 121, the annotated text stream 137, or a combination thereof from the server 204 over the network 106.The conference manager 162 plays out media frames of the media stream 109, the text stream 121, the annotated text stream 137, or a combination thereof, as described with reference to Figure 1. The interruption manager 164 trains the speech model 131, displays the avatar 135, or both, as described with reference to Figure 1.
[0065] In certain aspects, in response to detecting the network problem, the interruption manager 124 sends an interruption notification 119 to the device 104 indicating the interruption of the voice audio stream 111, refrains from sending (e.g., stops sending) subsequent media frames of the media stream 109 to the device 104, or both, until it detects that the network problem has been resolved (e.g., the interruption has ended). The interruption manager 124 sends the text stream 121, the metadata stream 123, or both, corresponding to the subsequent media frames to the device 104, as described with reference to FIG. 1. For example, the interruption manager 124 forwards the text stream 121, the metadata stream 123, or both, received from the device 102 to the device 104. In some examples, the interruption manager 124 sends the metadata stream 123, the text stream 121, or both, generated at the server 204 to the device 104. In a particular aspect, the interruption manager 124 selectively generates the metadata stream 123, the text stream 121, or both, in response to detecting an interruption in the voice audio stream 111 during a second operating mode of the server 204 (e.g., an interrupted data transmission mode).
[0066] In certain aspects, the interruption manager 164 detects an interruption in the voice audio stream 111 in a manner similar to that described with reference to FIG. 1 in response to receiving an interruption notification 119 from the interruption manager 124 (e.g., at the server 204), receiving the text stream 121, the metadata stream 123, or both when the server 204 is operating in a second operating mode (e.g., an interrupted data transmission mode), determining that an audio frame of the voice audio stream 111 is not received within a threshold duration of a last received audio frame of the voice audio stream 111, or a combination thereof. In certain aspects, the interruption manager 164 sends the interruption notification to the server 204. In certain aspects, the interruption manager 124 detects a network problem in response to receiving an interruption notification from the device 104. The interruption manager 124 sends the text stream 121, the metadata stream 123, or both, corresponding to a subsequent media frame, to the device 104 as described with reference to FIG. 1.
[0067] In response to detecting an interruption, the interruption manager 164 provides the text stream 121, the metadata stream 123, the annotated text stream 137, or a combination thereof, to the text-to-speech converter 166. The text-to-speech converter 166 generates a synthetic speech audio stream 133 by using the speech model 131 to perform text-to-speech conversion based on the text stream 121, the metadata stream 123, the annotated text stream 137, or a combination thereof, as described with reference to Fig. 1. In response to an interruption, the interruption manager 164 provides the synthetic speech audio stream 133 as an audio output 143 to the speaker 154, stops playing the speech audio stream 111, stops playing the video stream 113, displays the avatar 135, displays a particular representation of the avatar 135, displays the text stream 121, displays the annotated text stream 137, or a combination thereof, as described with reference to Fig. 1.
[0068] In response to detecting that the interruption has ended, the conference manager 122 resumes transmission of the voice audio stream 111, the video stream 113, or both, to the device 104. In certain aspects, in response to detecting that the interruption has ended during the second operating mode of the server 204 (e.g., the interrupted data transmission mode), the interruption manager 124 refrains from sending (e.g., ceases transmitting) the text stream 121, the metadata stream 123, or both, to the device 104.
[0069] In response to detecting that the interruption has ended, the conference manager 162 may refrain from generating the synthetic speech audio stream 133 based on the text stream 121, refrain from (e.g., stop) providing the synthetic speech audio stream 133 as audio output 143 to the speaker 154, resume playing the speech audio stream 111 as audio output 143 to the speaker 154, resume providing the video stream 113 to the display device 156, stop or adjust the display of the avatar 135, refrain from providing the text stream 121 to the display device 156, refrain from providing the annotated text stream 137 to the display device 156, or any combination thereof.
[0070] In this manner, system 200 reduces (e.g., eliminates) information loss during interruptions in voice audio stream 111 during online conferences with legacy devices (e.g., device 102 that does not include an interruption manager). For example, if a network problem prevents voice audio stream 111 from being received by device 104 but text can be received by device 104, user 144 continues to receive audio corresponding to user 142's voice (e.g., synthesized voice audio stream 133), text (e.g., text stream 121, annotated text stream 137, or both), or a combination thereof.
[0071] In certain aspects, the server 204 may be closer (e.g., fewer network hops) to the device 104, and sending the text stream 121, the metadata stream 123, or both from the server 204 (e.g., instead of from the device 102) may conserve overall network resources. In certain aspects, the server 204 may have access to network information that may be useful in successfully sending the text stream 121, the metadata stream 123, or both to the device 104. As an example, the server 204 initially sends the media stream 109 over a first network link. The server 204 detects a network problem and sends the text stream 121, the metadata stream 123, or both using a second network link that appears available to accept the text transmission based at least in part on a determination that the first network link is unavailable or not functioning.
[0072] 3A, there is shown an example of a GUI 145. In certain aspects, the GUI 145 is generated by the system 100 of FIG 1, the system 200 of FIG 2, or both.
[0073] The GUI 145 includes a video display 306, an avatar 135, and a training indicator (TI) 304. For example, the GUI generator 168 generates the GUI 145 during the initiation of an online conference. A video stream 113 (e.g., an image of a user 142 (e.g., Jill Pratt)) is displayed via the video display 306.
[0074] The training indicator 304 indicates the training level of the voice model 131 (e.g., 0% or not trained). For example, the training indicator 304 indicates that the voice model 131 has not been custom trained. In certain aspects, the representation of the avatar 135 (e.g., no color) also indicates the training level. In certain aspects, the representation of the avatar 135 indicates that no synthetic voice is being output. For example, the GUI 145 does not include a synthetic voice indicator such as that further described with reference to FIG. 3C.
[0075] In certain implementations, if an interruption occurs prior to custom training of the voice model 131 and the text-to-speech converter 166 uses the voice model 131 (e.g., a non-customized generic voice model) to generate the synthetic voice audio stream 133, the synthetic voice audio stream 133 corresponds to an audio voice having generic voice characteristics that may differ from the voice characteristics of the user 142. In certain aspects, the voice model 131 is initialized using a generic voice model associated with demographic data of the user 142. In this aspect, the synthetic voice audio stream 133 corresponds to generic voice characteristics that match the demographic data of the user 142 (e.g., age, gender, regional accent, etc.).
[0076] 3B, there is shown an example of a GUI 145. In certain aspects, the GUI 145 is generated by the system 100 of FIG. 1, the system 200 of FIG. 2, or both.
[0077] In certain examples, the GUI generator 168 updates the GUI 145 during the online conference. The training indicator 304 indicates a second training level (e.g., 20% or partially trained) of the voice model 131. For example, the training indicator 304 indicates that the voice model 131 is custom trained or partially custom trained. In certain aspects, the (e.g., partially colored) representation of the avatar 135 also indicates the second training level. In certain aspects, the representation of the avatar 135 indicates that no synthetic voice is being output. For example, the GUI 145 does not include a synthetic voice indicator.
[0078] In a particular implementation, when an interruption occurs after partial custom training of the voice model 131 and the text-to-speech converter 166 uses the voice model 131 (e.g., a partially customized voice model) to generate the synthetic voice audio stream 133, the synthetic voice audio stream 133 corresponds to an audio voice having voice characteristics that have some similarity to the voice characteristics of the user 142.
[0079] 3C, there is shown an example of a GUI 145. In certain embodiments, the GUI 145 is generated by the system 100 of FIG. 1, the system 200 of FIG. 2, or both.
[0080] In certain examples, the GUI generator 168 updates the GUI 145 in response to the interruption. The training indicator 304 indicates a third training level of the voice model 131 (e.g., 100% or training is complete). For example, the training indicator 304 indicates that the voice model 131 is being custom trained or that custom training is complete (e.g., a threshold level has been reached). In certain aspects, the (e.g., fully colored) representation of the avatar 135 also indicates the third training level. In certain aspects, the representation of the avatar 135 indicates that synthetic speech is being output. For example, the GUI 145 includes a synthetic speech indicator 398 displayed as part of or with the avatar 135 to indicate that the speech being played out is synthetic speech.
[0081] In the example of FIG. 3C, an interruption occurs after custom training of the voice model 131, and the text-to-speech converter 166 uses the voice model 131 (e.g., a customized voice model) to generate a synthetic voice audio stream 133, such that the synthetic voice audio stream 133 corresponds to an audio voice having voice characteristics similar to those of the user 142.
[0082] In response to the interruption, the interruption manager 164 stops outputting the video stream 113. For example, the video display 306 indicates that outputting the video stream 113 has been stopped due to an interruption (e.g., a network problem). The GUI 145 includes a text display 396. For example, the interruption manager 164 in response to the interruption outputs the text stream 121 via the text display 396.
[0083] In certain aspects, the text stream 121 is displayed in real time so that the user 144 can continue to participate in the conversation. For example, the user 144 can speak a response to the user 142 after reading on the text display 396 what the user 142 said. In certain aspects, if a network problem prevents a voice audio stream corresponding to the voice of the user 144 from being received by the device 102, the interruption manager 124 can display a text stream corresponding to the voice of the user 144 at the device 102. In this manner, one or more participants in an online conference can receive a text stream or a voice audio stream corresponding to the voice of the other participants.
[0084] Referring to Figure 4A, a diagram of an exemplary embodiment of the operation of system 100 of Figure 1 or system 200 of Figure 2 is shown and generally designated 400. The timing and operations shown in Figure 4A are illustrative and not limiting. In other embodiments, additional or fewer operations may be performed and the timing may differ.
[0085] Diagram 400 illustrates the timing of transmission of media frames of media stream 109 from device 102. In a particular aspect, media frames of media stream 109 are transmitted from device 102 to device 104 as described with reference to Figure 1. In an alternative aspect, media frames of media stream 109 are transmitted from device 102 to server 204 and from server 204 to device 102 as described with reference to Figure 2.
[0086] The device 102 transmits a media frame (FR) 410 of the media stream 109 at a first transmission time. The device 104 receives the media frame 410 at a first reception time and provides the media frame 410 for playback at a first playback time. In a particular example, the conference manager 162 buffers the media frame 410 during a first buffer interval between the first reception time and the first playback time. In a particular aspect, the media frame 410 includes a first portion of the video stream 113 and a first portion of the voice audio stream 111. The conference manager 162 outputs the first portion of the voice audio stream 111 as a first portion of the audio output 143 to the speaker 154 and the first portion of the video stream 113 to the display device 156 at the first playback time.
[0087] The device 102 (or the server 204) is expected to transmit the media frame 411 at a second expected transmission time. The device 104 is expected to receive the media frame 411 at a second expected reception time. The interruption manager 164 of the device 104 detects an interruption in the voice audio stream 111 in response to determining that a media frame of the media stream 109 is not received within a reception threshold duration of the first reception time. For example, the interruption manager 164 determines the second time based on the first reception time and the reception threshold duration (e.g., the second time=first reception time+reception threshold duration). The interruption manager 164 detects an interruption in the voice audio stream 111 in response to determining that a media frame of the media stream 109 is not received between the first reception time and the second time. The second time is after the second expected reception time of the media frame 411 and prior to the expected play time of the media frame 411. For example, the second time is during an expected buffer interval of the media frame 411.
[0088] The device 102 (or the server 204) detects the interruption in the voice audio stream 111 as described with reference to FIGS. 1-2. In response to the interruption in the voice audio stream 111, the interruption manager 124 (of the device 102 or the server 204) sends the text stream 121 corresponding to subsequent media frames (e.g., the set of media frames 491) to the device 104 until the interruption ends. In a particular aspect, the media frames 411 include a second portion of the video stream 113 and a second portion of the voice audio stream 111. The interruption manager 124 (or the conference manager 122) generates text 451 for the text stream 121 by performing speech-to-text conversion on the second portion of the voice audio stream 111 and sends the text 451 to the device 104.
[0089] The device 104 receives text 451 of the text stream 121 from the device 102 or the server 204, as described with reference to FIGS. 1-2. In response to the interruption, the interruption manager 164 begins playing the text stream 121 corresponding to subsequent media frames until the interruption ends. For example, the interruption manager 164 provides the text 451 to the display device 156 at a second playback time. In certain aspects, the second playback time is based on (e.g., is the same as) an expected playback time of the media frame 411.
[0090] In certain aspects, the conference manager 222 of FIG. 2 is unaware of the interruption and sends media frames 413 of the media stream 109 to the server 204. In certain aspects, the interruption manager 124 (of the device 102 of FIG. 1 or the server 204 of FIG. 2) stops sending media frames 413 to the device 104 in response to the interruption. In certain aspects, the media frames 413 include a third portion of the video stream 113 and a third portion of the voice audio stream 111. The interruption manager 124 generates text 453 based on the third portion of the voice audio stream 111. The interruption manager 124 sends the text 453 to the device 104.
[0091] The device 104 receives the text 453. The interruption manager 164, in response to the interruption, provides the text 453 to the display device 156 at a third play time. In certain aspects, the third play time is based on (e.g., is the same as) the expected play time of the media frame 413.
[0092] The interruption manager 124 (of the device 102 or the server 204) resumes transmission of subsequent media frames (e.g., next media frame 493) of the media stream 109 to the device 104 in response to the interruption ending, as described with reference to FIGS. 1-2. For example, the conference manager 122 transmits media frame 415 to the device 104. The interruption manager 164 resumes playback of the media stream 109 and stops playback of the text stream 121 in response to the interruption ending. In a particular aspect, the media frame 415 includes a fourth portion of the video stream 113 and a fourth portion of the voice audio stream 111. The conference manager 162 outputs the fourth portion of the voice audio stream 111 to the speaker 154 as part of the audio output 143 and outputs the fourth portion of the video stream 113 to the display device 156 at the fourth playback time.
[0093] As another example, conference manager 122 sends media frame 417 to device 104. In a particular aspect, media frame 417 includes a fifth portion of video stream 113 and a fifth portion of voice audio stream 111. Conference manager 162 outputs the fifth portion of voice audio stream 111 to speaker 154 as part of audio output 143 and outputs the fifth portion of video stream 113 to display device 156 at a fifth play time.
[0094] In this way, device 104 prevents information loss by playing text stream 121 during interruptions in media stream 109. Playback of media stream 109 resumes when the interruption ends.
[0095] Referring to Figure 4B, a diagram of an exemplary embodiment of the operation of the system 100 of Figure 1 or the system 200 of Figure 2 is shown and generally designated 490. The timing and operations shown in Figure 4B are for illustration purposes and not limitation. In other embodiments, additional or fewer operations may be performed and the timing may differ.
[0096] Diagram 490 illustrates the timing of transmission of media frames of media stream 109 from device 102. GUI generator 168 of FIG. 1 generates GUI 145 indicating a training level of avatar 135. For example, GUI 145 indicates that avatar 135 (e.g., voice model 131) is untrained or partially trained. Device 104 receives media frame 410 including a first portion of video stream 113 and a first portion of voice audio stream 111. Conference manager 162 outputs the first portion of voice audio stream 111 to speaker 154 as a first portion of audio output 143 and outputs the first portion of video stream 113 to display device 156 at a first playback time, as described with reference to FIG. 4A. Interruption manager 164 trains voice model 131 based on media frame 410 (e.g., first portion of voice audio stream 111), as described with reference to FIG. 1. The GUI generator 168 updates the GUI 145 to indicate the updated training level of the avatar 135 (eg, partially trained or fully trained).
[0097] The device 104 receives text 451 of the text stream 121 from the device 102 or the server 204, as described with reference to FIG. 4A. In response to the interruption, the interruption manager 164 stops playing the media stream 109, stops training the speech model 131, and starts playing the synthetic speech audio stream 133. For example, the interruption manager 164 generates synthetic speech frames 471 of the synthetic speech audio stream 133 based on the text 451. Illustratively, the interruption manager 164 provides the text 451 to the text-to-speech converter 166. The text-to-speech converter 166 uses the speech model 131 to perform text-to-speech conversion on the text 451 to generate synthetic speech frames (SFRs) 471. The interruption manager 164 provides the synthetic speech frames 471 as a second portion of the audio output 143 at a second playback time. The GUI generator 168 updates the GUI 145 to include a synthetic speech indicator 398 indicating that synthetic speech is being output. For example, GUI 145 shows avatar 135 speaking.
[0098] The device 104 receives the text 453, as described with reference to Figure 4A. The interruption manager 164, in response to the interruption, generates a synthesized speech frame 473 of the synthesized speech audio stream 133 based on the text 453. The interruption manager 164 provides the synthesized speech frame 473 as a third portion of the audio output 143 at a third playback time.
[0099] The interruption manager 124 (of the device 102 or the server 204) resumes sending subsequent media frames (e.g., next media frame 493) of the media stream 109 to the device 104 in response to the interruption ending, as described with reference to FIG. 4A. For example, the conference manager 122 sends media frame 415 to the device 104. The interruption manager 164 resumes playing the media stream 109, stops playing the synthetic speech audio stream 133, and resumes training of the speech model 131 in response to the interruption ending. The GUI generator 168 updates the GUI 145 to remove the synthetic speech indicator 398, which indicates that synthetic speech is not being output.
[0100] In a particular example, conference manager 162 plays out media frame 415 and media frame 417. Illustratively, media frame 415 includes a fourth portion of video stream 113 and a fourth portion of voice audio stream 111. Conference manager 162 outputs the fourth portion of voice audio stream 111 as a fourth portion of audio output 143 to speaker 154 and the fourth portion of video stream 113 to display device 156 at a fourth playback time. In a particular aspect, conference manager 162 outputs the fifth portion of voice audio stream 111 as a fifth portion of audio output 143 to speaker 154 and the fifth portion of video stream 113 to display device 156 at a fifth playback time.
[0101] In this way, device 104 prevents information loss by playing synthetic speech audio stream 133 during interruptions in media stream 109. Playback of media stream 109 resumes when the interruption ends.
[0102] 5, a system operable to handle voice audio stream interruptions is shown and generally designated 500. In certain aspects, the system 100 of FIG. 1 includes one or more components of the system 500.
[0103] The system 500 includes a device 502 coupled to a device 104 via a network 106. During operation, a conference manager 162 establishes an online conference with multiple devices (e.g., device 102 and device 502). For example, the conference manager 162 establishes an online conference of a user 144 with a user 142 of the device 102 and a user 542 of the device 502. The device 104 receives a media stream 109 (e.g., a voice audio stream 111, a video stream 113, or both) representing the voice, image, or both of the user 142 from the device 102 or the server 204 as described with reference to FIGS. 1-2. Similarly, the device 104 receives a media stream 509 (e.g., a second voice audio stream 511, a second video stream 513, or both) representing the voice, image, or both of the user 542 from the device 502 or a server (e.g., the server 204 or another server).
[0104] The conference manager 162 plays out the media stream 109 at the same time as it plays out the media stream 509, as will be further described with reference to FIG. 6A. For example, the conference manager 162 provides the video stream 113 to the display device 156 at the same time as it provides the second video stream 513 to the display device 156. To illustrate, the user 144 can view an image of the user 142 at the same time as it views an image of the user 542 during the online conference. As another example, the conference manager 162 provides the voice audio stream 111, the second voice audio stream 511, or both to the speaker 154 as the audio output 143. To illustrate, the user 144 can hear the voice of the user 142, the voice of the user 542, or both. In a particular aspect, the interruption manager 164 trains the voice model 131 based on the voice audio stream 111, as described with reference to FIG. 1. Similarly, the interruption manager 164 trains a second voice model for the user 542 based on the second voice audio stream 511 .
[0105] In a particular example, the device 104 continues to receive the media stream 509 during an interruption in the voice audio stream 111. The interruption manager 164 plays out the media stream 509 simultaneously with playing out the synthetic voice audio stream 133, the text stream 121, the annotated text stream 137, or a combination thereof, as further described with reference to FIG. 6C. For example, the interruption manager 164 generates the synthetic voice audio stream 133 and provides the second voice audio stream 511 simultaneously with providing the synthetic voice audio stream 133 to the speaker 154. As another example, the interruption manager 164 generates updates to the GUI 145, including the text stream 121 or the annotated text stream 137, and provides the second video stream 513 to the display device 156 simultaneously with providing the updates to the GUI 145 to the display device 156. In this manner, the user 144 can keep up with the conversation between the user 142 and the user 542 during an interruption in the voice audio stream 111.
[0106] In certain aspects, the interruption of the media stream 509 overlaps with the interruption of the voice audio stream 111. The interruption manager 164 receives the second text stream, the second metadata stream, or both, corresponding to the second voice audio stream 511. In certain aspects, the interruption manager 164 generates a second annotated text stream based on the second text stream, the second metadata stream, or both. The interruption manager 164 generates a second synthetic voice audio stream by using a second voice model to perform text-to-speech conversion based on the second text stream, the second metadata stream, the second annotated text stream, or a combination thereof. The interruption manager 164 plays out the second voice audio stream 511 to the speaker 154 simultaneously with playing out the synthetic voice audio stream 133. In certain aspects, interruption manager 164 plays out text stream 121, annotated text stream 137, or both, simultaneously with playing out the second text stream, the second annotated text stream, or both to display device 156. In this manner, user 144 can keep up with the conversation between user 142 and user 542 during interruptions in voice audio stream 111 and second voice audio stream 511.
[0107] In this manner, system 500 reduces (e.g., eliminates) information loss during interruptions of one or more voice audio streams (e.g., voice audio stream 111, second voice audio stream 511, or both) during an online conference with multiple users. For example, if a network problem prevents one or more voice audio streams from being received by device 104, but text can be received by device 104, user 144 continues to receive the voice of user 142 and the audio, text, or a combination thereof corresponding to the voice of user 542.
[0108] 6A, there is shown an example of the GUI 145. In a particular embodiment, the GUI 145 is generated by the system 500 of FIG.
[0109] The GUI 145 includes video displays, avatars, training indicators, or combinations thereof for multiple participants of the online conference. For example, the GUI 145 includes a video display 306, an avatar 135, a training indicator 304, or combinations thereof for the user 142, as described with reference to FIG. 3A. The GUI 145 also includes a video display 606, an avatar 635, a training indicator (TI) 604, or combinations thereof for the user 542. For example, the GUI generator 168 generates the GUI 145 during the start of the online conference. Concurrent with the display of the video stream 113 (e.g., an image of the user 142 (e.g., Jill P.)) via the video display 306, a second video stream 513 (e.g., an image of the user 542 (e.g., Emily F.)) of the media stream 509 is displayed via the video display 606.
[0110] The training indicator 304 indicates the training level of the voice model 131 (e.g., 0% or untrained), and the training indicator 604 indicates the training level of the second voice model (e.g., 10% or partially trained). The training levels of the voice models may be different if one user speaks more than the other user or if one user's voice contains a wider variety of sounds (e.g., higher model coverage).
[0111] In certain aspects, the representations of avatar 135 (e.g., uncolored) and avatar 635 (e.g., partially colored) also indicate the training level of the respective voice models. In certain aspects, the representations of avatar 135 and avatar 635 indicate that no synthetic voice is being output. For example, GUI 145 does not include any synthetic voice indicator.
[0112] In certain implementations, when an interruption occurs in receiving the media stream 109, the text-to-speech converter 166 generates the synthetic speech audio stream 133 using the voice model 131 (e.g., a non-customized generic voice model). When an interruption occurs in receiving the media stream 509, the text-to-speech converter 166 generates the second synthetic speech audio stream using the second voice model (e.g., a partially customized voice model). In certain aspects, the interruption manager 164 initializes the second voice model based on a second generic voice model that is distinct from the first generic voice model used to initialize the voice model 131 such that the synthetic voice of the user 142 is distinguishable from the synthetic voice of the user 542 when an interruption occurs prior to training (or full training) of the voice model 131 and the second voice model. In certain aspects, the voice model 131 is initialized using the first generic voice model associated with demographic data of the user 142, and the second voice model is initialized using the second generic voice model associated with demographic data of the user 542.
[0113] 6B, there is shown an example of the GUI 145. In certain aspects, the GUI 145 is generated by the system 500 of FIG.
[0114] In a particular example, the GUI generator 168 updates the GUI 145 during the online conference. For example, the training indicator 304 indicates a second training level of the speech model 131 (e.g., 20% or partially trained) and a second training level of the second speech model (e.g., 100% or fully trained).
[0115] 6C, there is shown an example of a GUI 145. In certain embodiments, the GUI 145 is generated by the system 500 of FIG.
[0116] In a particular example, the GUI generator 168 updates the GUI 145 in response to an interruption in the receipt of the media stream 109. The training indicator 304 indicates a third training level of the voice model 131 (e.g., 55% or partially trained) and the training indicator 604 indicates a third training level of the second voice model (e.g., 100% or fully trained). In a particular aspect, the representation of the avatar 135 indicates that synthetic voice is being output. For example, the GUI 145 includes a synthetic voice indicator 398. The representation of the avatar 635 indicates that synthetic voice is not being output to the user 542. For example, the GUI 145 does not include a synthetic voice indicator associated with the avatar 635.
[0117] In response to the interruption, the interruption manager 164 stops outputting the video stream 113. For example, the video display 306 indicates that the output of the video stream 113 has been stopped due to an interruption (e.g., a network problem). In response to the interruption, the interruption manager 164 outputs the text stream 121 via the text display 396.
[0118] In certain aspects, the text stream 121 is displayed in real time so that the user 144 can keep up with and participate in the conversation. For example, the user 144 can hear from the synthesized speech audio stream 133 and / or read on the text display 396 that the user 142 has made a first utterance (e.g., "I hope you have something to celebrate"). The user 144 can hear a response from the user 542 in a second speech audio stream of the media stream 509 output by the speaker 154. The user 144 can hear from the synthesized speech audio stream 133 and / or read on the text display 396 that the user 142 has made a second utterance (e.g., "That's very interesting. I'm glad you enjoyed it"). In this way, the user 144 can hear audio from the synthesized speech audio stream and / or read text in the text stream for one or more participants in the online conference while receiving a media stream for one or more other participants in the online conference.
[0119] Referring to Figure 7A, a diagram of an exemplary embodiment of the operation of the system 500 of Figure 5 is shown and generally designated 700. The timing and operations shown in Figure 7A are for illustration purposes and not limitation. In other embodiments, additional or fewer operations may be performed and the timing may differ.
[0120] Diagram 700 illustrates the timing of transmission of media frames of media stream 109 from device 102 and media stream 509 from device 502. In a particular aspect, media frames of media stream 109 are transmitted from device 102 or server 204 to device 104 as described with reference to Figures 1-2. Similarly, media frames of media stream 509 are transmitted from device 502 or a server (e.g., server 204 or another server) to device 104.
[0121] Device 104 receives media frames 410 of media stream 109 and media frames 710 of media stream 509 and provides media frames 410 and 710 for playback. For example, conference manager 162 outputs a first portion of voice audio stream 111 (e.g., as shown by media frame 410) and a first portion of a second voice audio stream (e.g., as shown by media frame 710) to speaker 154 as audio output 143, outputs a first portion of video stream 113 (e.g., as shown by media frame 410) via video display 306, and outputs a first portion of a second video stream (e.g., as shown by media frame 710) via video display 606, as described with reference to FIG.
[0122] Device 104 receives text 451 (corresponding to media frame 411) of text stream 121 during the interruption in media stream 109, as described with reference to FIG. 4A. Device 104 receives media frame 711 of media stream 509. In response to the interruption, interruption manager 164 begins playing text stream 121 corresponding to subsequent media frames of media stream 109, simultaneously with the playback of media stream 509, until the interruption ends. For example, interruption manager 164 provides text 451 (e.g., indicated by media frame 411) to display device 156 simultaneously with providing media frame 711 for playback.
[0123] Device 104 receives text 453 (corresponding to media frame 413) of text stream 121 during a break in media stream 109, as described with reference to Figure 4A. Device 104 receives media frame 713 of media stream 509. Break manager 164 provides text 453 to display device 156 at the same time as providing media frame 713 for playback.
[0124] 4A, in response to the interruption ending, interruption manager 164 resumes playback of media stream 109 and stops playback of text stream 121. Conference manager 162 receives and plays media frames 415 and 715. Similarly, conference manager 162 receives and plays media frames 417 and 717.
[0125] In this way, device 104 prevents information loss by playing text stream 121 during interruptions in media stream 109 simultaneously with playing media stream 509. Playback of media stream 109 resumes when the interruption ends.
[0126] Referring to Figure 7B, a diagram of an example embodiment of the operation of the system 500 of Figure 5 is shown and generally designated 790. The timing and operations shown in Figure 7B are for illustration purposes and not limitation. In other embodiments, additional or fewer operations may be performed and the timing may differ.
[0127] Diagram 790 illustrates the timing of transmission of media frames of media stream 109 from device 102 and media stream 509 from device 502. GUI generator 168 of FIG. 1 generates GUI 145 indicating the training level of avatar 135 and the training level of avatar 635. For example, GUI 145 indicates that avatar 135 (e.g., voice model 131) is not trained and avatar 635 (e.g., second voice model) is partially trained. Device 104 receives and plays media frames 410 and media frames 710. Interruption manager 164 trains voice model 131 based on media frames 410 and trains second voice model based on media frames 710, as described with reference to FIG. 4B. GUI generator 168 updates GUI 145 to indicate the updated training level of avatar 135 (e.g., partially trained) and the updated training level of avatar 635 (e.g., fully trained).
[0128] The device 104 receives the text 451 and media frames 711 of the text stream 121. The interruption manager 164 generates the synthetic speech frames 471 based on the text 451, as described with reference to FIG. 4B. The interruption manager 164 plays the synthetic speech frames 471 and the media frames 711. The GUI generator 168 updates the GUI 145 to include a synthetic speech indicator 398 that indicates to the user 142 that synthetic speech is being output. For example, the GUI 145 indicates that the avatar 135 is speaking. The GUI 145 does not include a synthetic speech indicator for the user 542 (e.g., the avatar 635 is not shown as speaking).
[0129] The device 104 receives the text 453 and the media frames 713. The interruption manager 164 generates the synthetic speech frames 473 based on the text 453, as described with reference to Figure 4B. The interruption manager 164 plays the synthetic speech frames 473 and the media frames 713.
[0130] The interruption manager 164, in response to the interruption ending, resumes playing the media stream 109, stops playing the synthetic speech audio stream 133, and resumes training of the speech model 131, as described with reference to Figure 4B. The GUI generator 168 updates the GUI 145 to remove the synthetic speech indicator 398, which indicates that synthetic speech is not being output.
[0131] In a particular example, conference manager 162 receives and plays out media frame 415 and media frame 715. As another example, conference manager 162 receives and plays out media frame 417 and media frame 717.
[0132] In this way, device 104 prevents information loss by simultaneously playing out media stream 509 and playing synthetic speech audio stream 133 during interruptions in media stream 109. Playback of media stream 109 resumes when the interruption ends.
[0133] 8, a particular implementation of a method 800 for handling voice audio stream interruptions is shown. In a particular aspect, one or more operations of the method 800 are performed by the conference manager 162, the interruption manager 164, one or more processors 160, the device 104, the system 100, or a combination thereof of FIG.
[0134] The method 800 includes receiving a voice audio stream representative of a voice of a first user during an online conference, at 802. For example, the device 104 of FIG. 1 receives a voice audio stream 111 representative of a voice of a user 142 during an online conference, as described with reference to FIG.
[0135] The method 800 also includes receiving a text stream representative of the speech of the first user, at 804. For example, the device 104 of FIG. 1 receives the text stream 121 representative of the speech of the user 142, as described with reference to FIG.
[0136] The method 800 further includes selectively generating an output based on the text stream in response to an interruption in the voice audio stream, at 806. For example, the interruption manager 164 of Figure 1 selectively generates the synthesized voice audio stream 133 based on the text stream 121 in response to an interruption in the voice audio stream 111, as described with reference to Figure 1. In a particular implementation, the interruption manager 164 selectively outputs the text stream 121, the annotated text stream 137, or both, in response to an interruption in the voice audio stream 111, as described with reference to Figure 1.
[0137] The method 800 ameliorates, and thus reduces (e.g., eliminates), information loss during interruptions in the voice audio stream 111 during an online conference. For example, if a network problem prevents the voice audio stream 111 from being received by the device 104, but text can be received by the device 104, the user 144 continues to receive audio corresponding to the voice of the user 142 (e.g., the synthesized voice audio stream 133), text (e.g., the text stream 121, the annotated text stream 137, or both), or a combination thereof.
[0138] The method 800 of Figure 8 may be implemented by a field programmable gate array (FPGA) device, an application specific integrated circuit (ASIC), a processing unit such as a central processing unit (CPU), a DSP, a controller, another hardware device, a firmware device, or any combination thereof. As an example, the method 800 of Figure 8 may be performed by a processor executing instructions such as those described with reference to Figure 18.
[0139] 9 illustrates one implementation 900 of device 104 as an integrated circuit 902 that includes one or more processors 160. The integrated circuit 902 also includes inputs 904 (e.g., one or more bus interfaces) to allow input data 928 (e.g., voice audio stream 111, video stream 113, media stream 109, interrupt notification 119, text stream 121, metadata stream 123, media stream 509, or combinations thereof) to be received for processing. The integrated circuit 902 also includes outputs 906 (e.g., bus interfaces) to allow transmission of output signals (e.g., voice audio stream 111, synthesized voice audio stream 133, audio output 143, video stream 113, text stream 121, annotated text stream 137, GUI 145, or combinations thereof). The integrated circuit 902 enables implementations that handle voice audio stream interruptions as components in a system, such as the mobile phone or tablet shown in FIG. 10, the headset shown in FIG. 11, the wearable electronic device shown in FIG. 12, the voice-controlled speaker system shown in FIG. 13, the camera shown in FIG. 14, the virtual reality or augmented reality headset shown in FIG. 15, or the vehicle shown in FIG. 16 or FIG. 17.
[0140] 10 illustrates an implementation 1000 in which the device 104 includes a mobile device 1002, such as a phone or tablet, as an illustrative and non-limiting example. The mobile device 1002 includes a microphone 1010, a speaker 154, and a display screen 1004. Components of one or more processors 160, including a conference manager 162, an interruption manager 164, a GUI generator 168, or a combination thereof, are integrated into the mobile device 1002 and are shown using dashed lines to indicate internal components that are generally not visible to a user of the mobile device 1002. In a particular example, the conference manager 162 outputs a voice audio stream 111 or the interruption manager 164 outputs a synthetic voice audio stream 133, and the voice audio stream 111 or the synthetic voice audio stream 133 are then processed to perform one or more operations on the mobile device 1002, such as to launch a graphical user interface (e.g., via an integrated "smart assistant" application) or to otherwise display other information associated with the user's voice on the display screen 1004.
[0141] 11 illustrates an implementation 1100 in which the device 104 includes a headset device 1102. The headset device 1102 includes a speaker 154, a microphone 1110, or both. One or more components of the processor 160, including a conference manager 162, an interruption manager 164, or both, are integrated into the headset device 1102. In a particular example, the conference manager 162 outputs a voice audio stream 111 or the interruption manager 164 outputs a synthetic voice audio stream 133, and the voice audio stream 111 or the synthetic voice audio stream 133 can cause the headset device 1102 to perform one or more operations in the headset device 1102 to transmit audio data corresponding to a user voice to a second device (not shown) for further processing.
[0142] 12 illustrates an implementation 1200 in which the device 104 includes a wearable electronic device 1202 depicted as a "smart watch." The conference manager 162, the interruption manager 164, the GUI generator 168, the speaker 154, the microphone 1210, or a combination thereof, are integrated into the wearable electronic device 1202. In a particular example, the conference manager 162 outputs the voice audio stream 111 or the interruption manager 164 outputs the synthetic voice audio stream 133, and the voice audio stream 111 or the synthetic voice audio stream 133 is then processed to perform one or more operations on the wearable electronic device 1202, such as to launch the GUI 145 or otherwise display other information associated with the user's voice on a display screen 1204 of the wearable electronic device 1202. To illustrate, the wearable electronic device 1202 may include a display screen configured to display notifications based on the user's voice detected by the wearable electronic device 1202. In a particular example, the wearable electronic device 1202 includes a haptic device that provides a haptic notification (e.g., vibrates) in response to detection of a user's voice. For example, the haptic notification may cause a user to look at the wearable electronic device 1202 to see a displayed notification indicating detection of a keyword spoken by the user. In this manner, the wearable electronic device 1202 may alert a user who is hearing impaired or who is wearing a headset that the user's voice has been detected.
[0143] FIG. 13 is an implementation 1300 in which the device 104 includes a wireless speaker and voice-activated device 1302. The wireless speaker and voice-activated device 1302 can have wireless network connectivity and is configured to perform assistant operations. One or more processors 160 including a conference manager 162, an interruption manager 164, or both, a speaker 154, a microphone 1310, or a combination thereof are included in the wireless speaker and voice-activated device 1302. In operation, in response to receiving a verbal command identified as a user voice in the voice audio stream 111 output by the conference manager 162 or in the synthesized voice audio stream 133 output by the interruption manager 164, the wireless speaker and voice-activated device 1302 can perform assistant operations, such as through execution of a voice activation system (e.g., an integrated assistant application). The assistant operations can include creating a calendar event, adjusting the temperature, playing music, turning on the lights, etc. For example, an assistant action is performed in response to receiving a command followed by a keyword or key phrase (e.g., "Hello, assistant").
[0144] 14 illustrates an implementation 1400 in which the device 104 includes a portable electronic device corresponding to a camera device 1402. The conference manager 162, the interruption manager 164, the GUI generator 168, the speaker 154, the microphone 1410, or a combination thereof, are included in the camera device 1402. In operation, in response to receiving a verbal command identified as a user voice in the voice audio stream 111 output by the conference manager 162 or in the synthesized voice audio stream 133 output by the interruption manager 164, the camera device 1402 can perform operations responsive to the verbal user command, such as, as illustrative examples, to adjust image or video capture settings, image or video playback settings, or image or video capture instructions.
[0145] 15 illustrates an implementation 1500 in which the device 104 includes a portable electronic device corresponding to a virtual reality, augmented reality, or mixed reality headset 1502. The conference manager 162, the interruption manager 164, the GUI generator 168, the speaker 154, the microphone 1510, or a combination thereof, are integrated into the headset 1502. User voice detection may be performed based on the voice audio stream 111 output by the conference manager 162 or the synthesized voice audio stream 133 output by the interruption manager 164. The visual interface device is placed in front of the user's eyes to enable the user to display an augmented reality or virtual reality image or scene while the headset 1502 is worn. In a particular example, the visual interface device is configured to display a notification indicating the user voice detected in the audio stream. In another example, the visual interface device is configured to display a GUI 145.
[0146] 16 illustrates an implementation 1600 in which the device 104 corresponds to or is integrated within a vehicle 1602 depicted as a manned or unmanned aerial device (e.g., a delivery drone). The conference manager 162, the interruption manager 164, the GUI generator 168, the speaker 154, the microphone 1610, or a combination thereof, are integrated into the vehicle 1602. User voice detection may be performed based on the voice audio stream 111 output by the conference manager 162 or the synthesized voice audio stream 133 output by the interruption manager 164, such as for delivery instructions from an authorized user of the vehicle 1602.
[0147] 17 shows another implementation 1700 in which the device 104 corresponds to or is integrated within a vehicle 1702, depicted as an automobile. The vehicle 1702 includes one or more processors 160, including a conference manager 162, an interruption manager 164, a GUI generator 168, or a combination thereof. The vehicle 1702 also includes a speaker 154, a microphone 1710, or both. User voice detection may be performed based on the voice audio stream 111 output by the conference manager 162 or the synthesized voice audio stream 133 output by the interruption manager 164. For example, the user voice detection may be used to detect a voice command from an authorized user of the vehicle 1702 (e.g., to start the engine or the heating). In certain implementations, in response to receiving a verbal command identified as a user voice in the voice audio stream 111 output by the conference manager 162 or in the synthetic voice audio stream 133 output by the interruption manager 164, the voice activation system of the vehicle 1702 initiates one or more actions of the vehicle 1702 based on one or more keywords (e.g., "unlock," "start the engine," "play music," "show weather," or another voice command) detected in the voice audio stream 111 or the synthetic voice audio stream 133, such as by providing feedback or information via the display 1720 or one or more speakers (e.g., speaker 154). In certain implementations, the GUI generator 168 provides information about the online conference (e.g., a call) to the display 1720. For example, the GUI generator 168 provides the GUI 145 to the display 1720.
[0148] 18, a block diagram of a particular example implementation of a device is shown, generally designated 1800. In various implementations, device 1800 may have more or fewer components than shown in FIG. 18. In an example implementation, device 1800 may correspond to device 104. In an example implementation, device 1800 may perform one or more of the operations described with reference to FIGS. 1-17.
[0149] In certain implementations, the device 1800 includes a processor 1806 (e.g., a central processing unit (CPU)). The device 1800 may include one or more additional processors 1810 (e.g., one or more DSPs). In certain aspects, the one or more processors 160 of FIG. 1 correspond to the processor 1806, the processor 1810, or a combination thereof. The processor 1810 may include a voice and music coder-decoder (codec) 1808, including a speech coder ("vocoder") encoder 1836, a vocoder decoder 1838, a conference manager 162, an interruption manager 164, a GUI generator 168, or a combination thereof. In certain aspects, the one or more processors 160 of FIG. 1 include the processor 1806, the processor 1810, or a combination thereof.
[0150] The device 1800 may include a memory 1886 and a codec 1834. The memory 1886 may include instructions 1856 executable by one or more additional processors 1810 (or processor 1806) to implement the functionality described with reference to the conference manager 162, the interruption manager 164, the GUI generator 168, or a combination thereof. In certain aspects, the memory 1886 stores program data 1858 used or generated by the conference manager 162, the interruption manager 164, the GUI generator 168, or a combination thereof. In certain aspects, the memory 1886 includes the memory 132 of FIG. 1. The device 1800 may include a modem 1840 coupled to an antenna 1842 via a transceiver 1850.
[0151] The device 1800 may include a display device 156 coupled to a display controller 1826. The speaker 154 and one or more microphones 1832 may be coupled to a codec 1834. The codec 1834 may include a digital-to-analog converter (DAC) 1802, an analog-to-digital converter (ADC) 1804, or both. In certain implementations, the codec 1834 can receive analog signals from the one or more microphones 1832, convert the analog signals to digital signals using an analog-to-digital converter 1804, and provide the digital signals to a voice and music codec 1808. The voice and music codec 1808 can process the digital signals, which may be further processed by the conference manager 162, the interruption manager 164, or both. In certain implementations, the voice and music codec 1808 can provide the digital signals to the codec 1834. The codec 1834 can convert the digital signals to analog signals using a digital-to-analog converter 1802, and provide the analog signals to the speaker 154.
[0152] In certain implementations, the device 1800 may be included in a system-in-package or system-on-chip device 1822. In certain implementations, the memory 1886, the processor 1806, the processor 1810, the display controller 1826, the codec 1834, the modem 1840, and the transceiver 1850 are included in the system-in-package or system-on-chip device 1822. In certain implementations, the input device 1830 and the power supply 1844 are coupled to the system-on-chip device 1822. Additionally, in certain implementations, the display device 156, the input device 1830, the speaker 154, the one or more microphones 1832, the antenna 1842, and the power supply 1844 are external to the system-on-chip device 1822, as shown in FIG. In particular implementations, each of the display device 156, the input device 1830, the speaker 154, the one or more microphones 1832, the antenna 1842, and the power source 1844 may be coupled to a component of the system-on-chip device 1822, such as an interface or controller.
[0153] The device 1800 may include a virtual assistant, a home appliance, a smart device, an Internet of Things (IoT) device, a communications device, a headset, a vehicle, a computer, a display device, a television, a game console, a music player, a radio, a video player, an entertainment unit, a personal media player, a digital video player, a camera, a navigation device, a smart speaker, a speaker bar, a mobile communications device, a smartphone, a cellular phone, a laptop computer, a tablet, a personal digital assistant, a digital video disc (DVD) player, a tuner, an augmented reality headset, a virtual reality headset, an aviation vehicle, a home automation system, a voice-activated device, a wireless speaker and a voice-activated device, a portable electronic device, an automobile, a computing device, a virtual reality (VR) device, a base station, a mobile device, or any combination thereof.
[0154] In relation to the described implementation, the apparatus includes a means for receiving a voice audio stream during an online conference, where the voice audio stream represents a voice of a first user. For example, the means for receiving a voice audio stream can correspond to the conference manager 162, the interruption manager 164, the one or more processors 160, the device 104, the system 100, the conference manager 122, the server 204, the system 200, the one or more processors 1810, the processor 1806, the voice and music codec 1808, the modem 1840, the transceiver 1850, the antenna 1842, the device 1800, one or more other circuits or components configured to receive a voice audio stream during an online conference, or any combination thereof.
[0155] The apparatus also includes means for receiving a text stream representative of the speech of the first user. For example, the means for receiving the text stream may correspond to the conference manager 162, the interruption manager 164, the text-to-speech converter 166, the one or more processors 160, the device 104, the system 100, the conference manager 122, the interruption manager 124, the server 204, the system 200, the one or more processors 1810, the processor 1806, the voice and music codec 1808, the modem 1840, the transceiver 1850, the antenna 1842, the device 1800, one or more other circuits or components configured to receive the text stream, or any combination thereof.
[0156] The apparatus further includes means for selectively generating an output based on the text stream in response to an interruption in the voice audio stream. For example, the means for selectively generating an output may correspond to the interruption manager 164 of FIG. 1, the text-to-speech converter 166, the GUI generator 168, the one or more processors 160, the device 104, the system 100, the interruption manager 124 of FIG. 2, the server 204, the system 200, the one or more processors 1810, the processor 1806, the voice and music codec 1808, the device 1800, one or more other circuits or components configured to selectively generate an output, or any combination thereof.
[0157] In some implementations, a non-transitory computer-readable medium (e.g., a computer-readable storage device such as memory 1886) includes instructions (e.g., instructions 1856) that, when executed by one or more processors (e.g., one or more processors 1810 or processor 1806), cause the one or more processors to receive a voice audio stream (e.g., voice audio stream 111) representing the voice of a first user (e.g., user 142) during an online conference. The instructions, when executed by the one or more processors, also cause the one or more processors to receive a text stream (e.g., text stream 121) representing the voice of the first user (e.g., user 142). The instructions, when executed by the one or more processors, further cause the one or more processors to selectively generate an output (e.g., synthesized voice audio stream 133, annotated text stream 137, or both) based on the text stream in response to an interruption of the voice audio stream.
[0158] Certain aspects of the disclosure are described below in a first set of interrelated clauses.
[0159] According to clause 1, a device for communications includes one or more processors, the one or more processors configured to receive a voice audio stream representing a voice of a first user during an online conference, receive a text stream representing the voice of the first user, and selectively generate an output based on the text stream in response to an interruption of the voice audio stream.
[0160] Clause 2 includes the device of clause 1, wherein the one or more processors are configured to detect the interruption in response to determining that an audio frame of the voice audio stream has not been received within a threshold duration of a last received audio frame of the voice audio stream.
[0161] Clause 3 includes the device of clause 1, wherein the one or more processors are configured to detect an interruption in response to receiving the text stream.
[0162] Clause 4 includes the device of clause 1, wherein the one or more processors are configured to detect the interruption in response to receiving the interruption notification.
[0163] Clause 5 includes the device of any of clauses 1-4, wherein the one or more processors are configured to provide the text stream as an output to a display.
[0164] Clause 6 includes the device of any of clauses 1-5, wherein the one or more processors are further configured to receive a metadata stream indicating an intonation of the first user's voice, and annotate the text stream based on the metadata stream.
[0165] Clause 7 includes the device of any of clauses 1-6, wherein the one or more processors are further configured to perform text-to-speech conversion on the text stream to generate a synthetic speech audio stream, and to provide the synthetic speech audio stream as an output to a speaker.
[0166] Clause 8 includes the device of clause 7, wherein the one or more processors are further configured to receive a metadata stream indicating an intonation of the first user's voice, and the text-to-speech conversion is based on the metadata stream.
[0167] Clause 9 includes the device of clause 7, wherein the one or more processors are further configured to display an avatar concurrently with providing the synthesized speech audio stream to the speaker.
[0168] Clause 10 includes the device of clause 9, wherein the one or more processors are configured to receive media streams during the online conference, the media streams including a voice audio stream and a video stream of the first user.
[0169] Clause 11 includes the device of clause 10, wherein the one or more processors are configured to, in response to an interruption, stop playing the voice audio stream and stop playing the video stream.
[0170] Clause 12 includes the device of clause 10, wherein the one or more processors are configured to, in response to the interruption terminating, refrain from providing the synthetic voice audio stream to the speaker, refrain from displaying the avatar, resume playing the video stream, and resume playing the voice audio stream.
[0171] Clause 13 includes the device of clause 7, wherein the text-to-speech conversion is performed based on a speech model.
[0172] Clause 14 includes the device of clause 13, wherein the speech model corresponds to a generic speech model.
[0173] Clause 15 includes the device of clause 13 or clause 14, wherein the one or more processors are configured to update a speech model based on the speech audio stream prior to the interruption.
[0174] Clause 16 includes the device of any of clauses 1-15, wherein the one or more processors are configured to receive a second voice audio stream representing a voice of a second user during the online conference, and to provide the second voice audio stream to a speaker simultaneously with generating the output.
[0175] Clause 17 includes the device of any of clauses 1-16, wherein the one or more processors are configured to, in response to an interruption in the audio audio stream, stop playing the audio audio stream, and, in response to the interruption ending, refrain from generating output based on the text stream and resume playing the audio audio stream.
[0176] Certain aspects of the disclosure are described below in a second set of interrelated clauses.
[0177] According to clause 18, a method of communication includes receiving, at a device, a voice audio stream representing a voice of a first user during an online conference; receiving, at the device, a text stream representing the voice of the first user; and selectively generating, at the device, an output based on the text stream in response to an interruption of the voice audio stream.
[0178] Clause 19 includes the method of clause 18, further including the step of detecting an interruption in response to determining that an audio frame of the voice audio stream has not been received within a threshold duration of a last received audio frame of the voice audio stream.
[0179] Clause 20 includes the method of clause 18, further including the step of detecting an interruption in response to receiving the text stream.
[0180] Clause 21 includes the method of clause 18, further including the step of detecting the interruption in response to receiving the interruption notification.
[0181] Clause 22 includes the method of any of clauses 18-21, further including providing the text stream as output to a display.
[0182] Clause 23 includes the method of any of clauses 18-22, further including receiving a metadata stream indicative of an intonation of the first user's voice, and annotating the text stream based on the metadata stream.
[0183] Certain aspects of the disclosure are described below in a third set of interrelated clauses.
[0184] According to clause 24, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors, cause the one or more processors to receive an audio voice stream representing a voice of a first user during an online conference, receive a text stream representing the voice of the first user, and selectively generate an output based on the text stream in response to an interruption of the audio voice stream.
[0185] Clause 25 includes the non-transitory computer-readable storage medium of clause 24, the instructions which, when executed by one or more processors, cause the one or more processors to perform text-to-speech conversion on the text stream to generate a synthetic speech audio stream, and to provide the synthetic speech audio stream as an output to a speaker.
[0186] Clause 26 includes the non-transitory computer-readable storage medium of clause 25, the instructions, when executed by the one or more processors, cause the one or more processors to receive a metadata stream indicative of an intonation of the first user's voice, and the text-to-speech conversion is based on the metadata stream.
[0187] Clause 27 includes the non-transitory computer-readable storage medium of clause 25 or clause 26, the instructions which, when executed by the one or more processors, cause the one or more processors to display an avatar simultaneously with providing a synthesized voice audio stream to a speaker.
[0188] Clause 28 includes the non-transitory computer-readable storage medium of any of clauses 25-27, wherein the instructions, when executed by the one or more processors, cause the one or more processors to update a speech model based on the voice audio stream prior to the interruption, and text-to-speech conversion is performed based on the speech model.
[0189] Certain aspects of the disclosure are described below in a fourth set of interrelated clauses.
[0190] According to clause 29, the apparatus includes means for receiving an audio stream during an online conference, the audio audio stream representing a speech of a first user, means for receiving a text stream representing the speech of the first user, and means for selectively generating an output based on the text stream in response to an interruption of the audio audio stream.
[0191] Clause 30 includes the apparatus of clause 29, wherein the means for receiving the voice audio stream, the means for receiving the text stream, and the means for selectively generating the output are integrated into at least one of a virtual assistant, a home appliance, a smart device, an Internet of Things (IoT) device, a communications device, a headset, a vehicle, a computer, a display device, a television, a games console, a music player, a radio, a video player, an entertainment unit, a personal media player, a digital video player, a camera, or a navigation device.
[0192] Those skilled in the art will further appreciate that the various exemplary logical blocks, configurations, modules, circuits, and algorithmic steps described with respect to the implementations disclosed herein may be implemented as electronic hardware, computer software executed by a processor, or a combination of both. Various exemplary components, blocks, configurations, modules, circuits, and steps have been described above generally with respect to their functionality. Whether such functionality is implemented as hardware or as processor-executable instructions depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, and such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.
[0193] The steps of a method or algorithm described with respect to the implementations disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a random access memory (RAM), a flash memory, a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), a register, a hard disk, a removable disk, a compact disk read only memory (CD-ROM), or any other form of non-transitory storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from and write information to the storage medium. Alternatively, the storage medium may be integral to the processor. The processor and the storage medium may reside in an application specific integrated circuit (ASIC). The ASIC may reside in a computing device or a user terminal. Alternatively, the processor and the storage medium may reside as discrete components in a computing device or a user terminal.
[0194] The foregoing description of the disclosed aspects is provided to enable any person skilled in the art to make or use the disclosed aspects. Various modifications of these aspects will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein, but is to be accorded the widest possible scope consistent with the principles and novel features defined by the following claims. [Explanation of symbols]
[0195] 100 Systems 102 Devices 104 Devices 106 Network 109 Media Streams 111 Voice Audio Stream 113 Video Stream 119 Suspension Notice 120 processors 121 Text Stream 122 Conference Manager 123 Metadata Stream 124 Suspension Manager 131 Voice Model 132 Memory 133 Synthetic Speech Audio Stream 135 Avatar 137 Annotated Text Streams 142 users 143 Audio Output 144 users 145 GUI 150 Camera 151 Video Input 152 Microphone 153 Audio Input 154 Speakers 156 Display Devices 160 processors 162 Conference Manager 164 Suspension Manager 166 Text to Speech Converter 168 Graphical User Interface (GUI) Generator, GUI Generator 200 Systems 204 Server 222 Conference Manager 304 Training Indicator (TI), Training Indicator 306 Video Display 396 Text Display 398 Synthetic Speech Indicator 400 Figures 410 Media Frame (FR), Media Frame 411 Media Frame 413 Media Frame 415 Media Frame 417 Media Frame 451 Text 453 Text 471 Synthetic Speech Frames 473 Synthetic Speech Frames Set of 491 media frames 493 Next Media Frame 500 Systems 502 Devices 509 Media Stream 511 Secondary Audio Stream 513 Secondary Video Stream 542 users 604 Training Indicator (TI), Training Indicator 606 Video Display 635 Avatar 700 Figures 710 Media Frame 711 Media Frame 713 Media Frame 715 Media Frame 717 Media Frame 790 Figures 800 ways 900 Implementation 902 Integrated Circuits 904 Input section 906 Output section 928 Input Data 1000 Implementation form 1002 Mobile Devices 1004 Display Screen 1010 Microphone 1100 Mounting form 1102 Headset Device 1110 Microphone 1200 Implementation 1202 Wearable Electronic Devices 1204 Display Screen 1210 Microphone 1300 Implementation form 1302 Wireless Speakers and Voice Activated Devices 1310 Microphone 1400 Implementation 1402 Camera Device 1410 Microphone 1500 Implementation 1502 Virtual reality, augmented reality, or mixed reality headsets; headsets 1510 Microphone 1600 Implementation 1602 Vehicle 1610 Microphone 1700 Mounting Form 1702 Vehicle 1710 Microphone 1720 Display 1800 devices 1802 Digital to Analog Converter (DAC), Digital to Analog Converter 1804 Analog-to-Digital Converter (ADC), Analog-to-Digital Converter 1806 Processor 1808 Speech and Music Codec Decoders (Codecs) 1810 Processor 1822 System in package or system on chip device, system on chip device 1826 Display Controller 1830 Input Devices 1832 Microphone 1834 Codec 1836 Voice Coder ("Vocoder") Encoder 1838 Vocoder Decoder 1840 Modem 1842 Antenna 1844 power supply 1850 Transceiver 1856 command 1858 Program Data 1886 Memory
Claims
1. A communication device, comprising: one or more processors, the one or more processors: receiving a voice audio stream representing a voice of a first user during an online conference; receiving a text stream representing the speech of the first user; receiving a metadata stream indicative of an intonation of the voice of the first user; selectively generating an output based on the text stream in response to an interruption in the voice audio stream; annotating the text stream based on the metadata stream; A device configured to:
2. 2. The device of claim 1, wherein the one or more processors are configured to detect the interruption in response to determining that an audio frame of the voice audio stream was not received within a threshold duration of a last received audio frame of the voice audio stream.
3. The one or more processors, Detecting the interruption in response to receiving an interruption notification; and / or providing said text stream as said output to a display; The device of claim 1 , configured to:
4. The one or more processors, performing text-to-speech conversion on the text stream to generate a synthetic speech audio stream; providing said synthetic speech audio stream as said output to a speaker; The device of claim 1 , further configured to:
5. 5. The device of claim 4, wherein the one or more processors are further configured to receive a metadata stream indicating an intonation of the voice of the first user, and wherein the text-to-speech conversion is based on the metadata stream.
6. 5. The device of claim 4, wherein the one or more processors are further configured to display an avatar simultaneously with providing the synthesized speech audio stream to the speaker.
7. 7. The device of claim 6, wherein the one or more processors are configured to receive media streams during the online conference, the media streams including the voice audio stream and video stream of the first user.
8. the one or more processors, in response to the interruption, stopping playback of the voice audio stream; stopping playback of said video stream; The device of claim 7 , configured to:
9. In response to the one or more processors terminating the interruption, refraining from providing the synthesized speech audio stream to the speaker; refraining from displaying said avatar; and resuming playback of the video stream; and resuming playback of the voice audio stream; The device of claim 7 , configured to:
10. The device of claim 4 , wherein the text-to-speech conversion is performed based on a speech model.
11. The device of claim 10 , wherein the speech model corresponds to a generic speech model.
12. 11. The device of claim 10, wherein the one or more processors are configured to update the voice model based on the voice audio stream prior to the interruption.
13. the one or more processors: receiving a second voice audio stream representative of a voice of a second user during the online conference; providing the second voice audio stream to a speaker simultaneously with generating the output; and / or in response to the interruption of the audio stream, ceasing playback of the audio stream; and In response to the interruption being terminated, refraining from generating said output based on said text stream; resuming playback of the voice audio stream; and The device of claim 1 , configured to:
14. A method of communication comprising: receiving, at the device, a voice audio stream representative of a voice of a first user during an online conference; receiving, at the device, a text stream representing the speech of the first user; receiving, at the device, a metadata stream indicative of an intonation of the voice of the first user; selectively generating, at the device, an output based on the text stream in response to an interruption of the voice audio stream; annotating the text stream based on the metadata stream; The method includes:
15. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to: receiving a voice audio stream representing a voice of a first user during an online conference; receiving a text stream representing the speech of the first user; receiving a metadata stream indicative of an intonation of the voice of the first user; selectively generating an output based on the text stream in response to an interruption in the voice audio stream; annotating the text stream based on the metadata stream; A non-transitory computer-readable storage medium that causes
