Maintaining connectivity in challenging network conditions with caption-assisted calls

Abstract

A system is provided for managing and coordinating communication between STT / TTS systems and these systems during online conferences, and for mitigating connectivity issues that may occur during online conferences to provide a seamless and reliable conference experience with real-time captioning and / or presented audio. Initially, online conference communication is transmitted via a lossy connectionless protocol / channel. Then, in response to detected connectivity issues with one or more systems involved in the online conference (e.g., which may cause jitter or packet loss), instructions are dynamically generated and processed to enable one or more of the connected systems to utilize a more reliable connection / protocol (such as a connection-oriented protocol) to transmit and / or process the online conference content. Codecs are used at the system level when it is necessary to convert speech to text with associated speech attribute information and to convert text to speech.

Description

Background Technology

[0001] Computers and computing systems influence virtually every aspect of modern life. For example, computers are commonly involved in work, leisure, healthcare, transportation, entertainment, home management, and so on.

[0002] Computers are configured with hardware and software interfaces to facilitate direct user interaction. These interfaces include, but are not limited to, keyboards, mice, touchpads, touchscreens, cameras, and microphones, which can receive and process user input, including voice and other audio input. Output interfaces include, but are not limited to, display devices, speakers, etc., which can process and present output, including text and audio / speech output.

[0003] Software interfaces can also be used to process and present various user inputs and outputs. Examples of software user interfaces include graphical user interfaces, text-based command-line user interfaces, function key or hotkey user interfaces, etc.

[0004] The functionality and utility of a computing system can often be enhanced by interconnecting it with other computing systems via network connections. Network connections can include, but are not limited to, wired or wireless Ethernet connections, cellular connections, or even computer-to-computer connections via serial, parallel, USB, or other connections. These connections allow the computing system to access services on other computing systems and to receive application data from other computing systems quickly and efficiently.

[0005] The interconnection of computing systems has facilitated distributed computing systems, such as those referred to as "cloud" computing systems. In this description, "cloud computing" can refer to a system or resource that enables ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, services, etc.), which can be provisioned and distributed with reduced management effort or service provider interaction.

[0006] The cloud model can consist of various features (e.g., on-demand self-service, broad network access, resource pools, rapid elasticity, measurement services, etc.), service models (e.g., Software as a Service (“SaaS”), Platform as a Service (“PaaS”), Infrastructure as a Service (“IaaS”)), and deployment models (e.g., private cloud, community cloud, public cloud, hybrid cloud, etc.).

[0007] Some existing applications (such as Microsoft's Teams app) facilitate collaboration among different users through online platforms that connect the computing devices of online meeting participants via various types of network connections, including distributed network connections (which may include the cloud). Online meeting content transmitted during a meeting may include audio, video, and text data encoded in data packets, which are transmitted as electronic communications between the online participants' computing systems via network connections.

[0008] In some instances, communication between the computing systems of online meeting participants is routed via a server over one or more network channels configured to use different protocols. For example, some electronic communications used for video and audio content are often transmitted via UDP (User Datagram Protocol). UDP is generally considered a lossy and connectionless protocol. Other electronic communications (such as text) are typically transmitted via connection-based protocols (such as TCP (Transmission Control Protocol), which is generally considered to have less loss than UDP.

[0009] For audio and video, UDP is generally preferred over TCP because UDP is faster than TCP. Specifically, audio transmission over UDP (e.g., approximately 10-60 Kbit / s) is an order of magnitude faster than text transmission over TCP (e.g., approximately 0.1 Kbit / s). On the other hand, especially for text, TCP is sometimes preferred over UDP because it is generally more reliable. For example, the TCP protocol ensures that its data packets are both received by (multiple) receiving devices and correctly ordered by (multiple) receiving devices. In contrast, the UDP protocol does not do this. Systems that transmit packets via UDP do not track whether their transmitted data packets are ultimately received by the receiving device, nor do they track the order in which data packets are received or presented at the receiving device.

[0010] When connectivity issues exist (such as changes in available bandwidth, signal interruptions, or other problems negatively affecting the delivery of transmitted packets), the presentation of input media at (multiple) receiver / presentation devices will experience "jitter" or disjointed data presentation. Disjointed presentation refers to the presentation of data that is omitted or skipped due to "lossy" connectivity. "Jitter" refers to the presentation of data packets in reverse order. It will be understood that jitter and lossy connectivity issues can have a very negative impact on online meetings by causing some words in the online content to be omitted or presented inaudibly and / or in reverse order.

[0011] For at least these reasons, existing systems facilitating online meetings currently face challenges related to providing reliable communication between meeting participants when connected user systems suffer from challenging network conditions (e.g., poor connectivity). Furthermore, even when systems are configured to use captions with the system for online meetings, these existing systems may still experience degraded communication in live meeting experiences when participants cannot hear each other correctly due to connectivity issues affecting audio transmission. Therefore, even for systems capable of using captions, there is a continued need and expectation to improve the reliability of transmitting online meeting content during online meetings under challenging network conditions.

[0012] Some known systems enable servers to independently detect network connectivity issues during video / audio sessions between teams and to transcribe received audio into text to be sent to the intended recipient. However, such systems do not address or resolve issues with audio communication to the server in the first instance, allowing audio to potentially be corrupted or degraded before it is received by the server, causing errors in the audio to persist during transcription.

[0013] Other known systems enable end nodes to independently determine whether changes should be made to their communication settings (e.g., to transcribe audio into text) to help facilitate more reliable communication that is sent to a server and routed to another user system. However, such systems cannot determine whether changes should be made based on conditions experienced at other remote systems, and cannot manage the configurations of other end nodes that may experience difficulties receiving / processing the data being sent.

[0014] Therefore, there remains a continued need and expectation for improved systems and technologies for managing network communications and for providing centralized mitigation control over such communications.

[0015] The subject matter claimed herein is not limited to embodiments that address any shortcomings or operate only in environments such as those described above. Rather, this background is provided merely to illustrate an exemplary area of ​​technology in which some of the embodiments described herein may be practiced. Summary of the Invention

[0016] Implementations are provided for managing dynamic changes in electronic communications between online conferencing systems in an effort to mitigate challenging network conditions.

[0017] In some embodiments, online conference communications (including audio and voice communications) are transmitted to one or more receiving computing systems via a lossy connectionless protocol / channel (e.g., UDP). Then, in response to detected connectivity issues (e.g., jitter or packet loss) in one or more client systems involved in the online conference, instructions are dynamically generated and processed to enable one or more codecs on the client systems to activate for transmitting and / or processing the online conference content using a more reliable connection / protocol (such as a connection-oriented protocol like TPC). These instructions may be generated by one of the server system and / or conference participant systems in response to the detection of connectivity issues. Connectivity issues may be detected automatically and / or in response to user input.

[0018] In some instances, in response to instructions(s), the client system that generates and transmits online meeting content will activate the appropriate STT (text-to-speech) codec / application to convert the online meeting content, which includes audio sound content, into text, and transmit the content (which is converted from audio to text) over a TCP channel, instead of as audio over a UDP channel previously used to transmit online meeting content.

[0019] Similarly, in some embodiments, and also in response to instructions(s), one or more receiving / presenting systems will activate / utilize a suitable TTS (speech-to-text) codec / application to convert text received over a TCP channel into audio sound content, and will present the audio through one or more speakers.

[0020] In other embodiments, the receiving system may use a suitable codec to process and present the text as text message content on a visual display, rather than, or otherwise, use the STT codec to convert the text back into audio content.

[0021] This summary is provided to introduce, in a simplified form, some concepts that will be further described in the detailed description below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0022] Additional features and advantages will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the teachings herein. The features and advantages of the invention may be realized and obtained by the means and combinations particularly pointed out in the appended claims. The features of the invention will become more fully apparent from the following description and the appended claims, or may be learned by practicing the invention as described below. Attached Figure Description

[0023] To describe in detail the ways in which the above and other advantages and features can be obtained, a more specific description of various embodiments will be given with reference to the accompanying drawings. It should be understood that these drawings depict only exemplary embodiments and are therefore not intended to limit the scope of the invention. The embodiments will be described and explained with additional features and details using the drawings, wherein:

[0024] Figure 1 The illustration depicts a computing environment in which a server facilitates electronic communication between different client computing systems and in which aspects of the present invention can be found or implemented.

[0025] Figure 2 The illustration depicts a computing environment in which a server facilitates electronic communication with a client computing system through different channels, including a first channel utilizing a connectionless protocol (such as UDP) for transmitting audio and video data and a second channel utilizing a connection-based protocol (such as TCP) for the transmitted text data.

[0026] Figure 3 The diagram illustrates flowcharts of various actions associated with a method executed by a client system for processing and transmitting online meeting content to other meeting participants via different communication channels through a server, and for responding to instructions to switch communication channels / protocols used for transmitting online meeting content during the online meeting.

[0027] Figure 4 The diagram illustrates flowcharts of various actions associated with a method executed by a client system for processing and transmitting online meeting content to other meeting participants via different communication channels through a server, as well as for generating instructions to activate codecs and trigger switching in communication channels / protocols used to process / transmit online meeting content during the online meeting.

[0028] Figure 5 The diagram illustrates a flowchart of various actions associated with a method performed by a client system for processing and presenting online meeting content received from other meeting participants via different communication channels through a server, and for responding to instructions to switch codecs used to process and / or present online meeting content during the online meeting.

[0029] Figure 6The diagram illustrates a composite flowchart of various actions associated with a method performed by both a client and a server system for processing and transmitting online meeting content via different communication channels through the server, as well as for dynamically generating instructions and / or responding to instructions, which activate codecs and trigger switching in communication channels / protocols used to process / transmit online meeting content during the online meeting. Detailed Implementation

[0030] The various embodiments described herein are used to facilitate and manage communication between systems that connect to online meetings and to improve reliability in the transmission of online meeting content between participating systems in an effort to mitigate challenging network conditions.

[0031] In some embodiments, in response to an instruction received from a server that detects a network connectivity problem, online conference content, including voice, is converted from an audio format to a text format. This network connectivity problem negatively affects the transmission of audio to one or more receiving devices, and thereby utilizes captioning assistance and orientation features to improve the reliability of online conference content transmission and prevent the loss of data being presented at the receiving system during online conferences subjected to challenging network conditions.

[0032] Some embodiments point to methods and systems implemented by client systems for dynamically changing codecs, channels, and / or protocols in response to detected connectivity problems / difficulties, which are used to process, transmit, and / or present online meeting content in response to instructions received from a server system during an online meeting.

[0033] Other embodiments point to server systems and methods for dynamically generating instructions to be sent to client systems in response to detected connectivity issues, and these instructions are operable to change the codecs, channels, and / or protocols used during online meetings.

[0034] The disclosed embodiments offer numerous technical advantages over existing systems and technologies used for managing online meetings and communications. Initially, it should be noted that the disclosed embodiments can be used to automatically switch between different types of communication channels and protocols used to transmit online content (particularly online voice content) in such a way as to improve the reliability of data transmission of the voice content in the event of negative connectivity problems, by converting audio to text and transmitting it over more reliable channels and protocols (e.g., TCP) (e.g., audio packets transmitted via UDP), so that data packets are not lost or out of order due to connectivity problems (e.g., poor connection, terminals in the connection, bandwidth limitations, etc.), while also facilitating faster communication over UDP for audio when negative connectivity problems are not detected or present.

[0035] In some embodiments, the disclosed invention facilitates further technical advantages over existing systems by providing centralized coordination and control of the activation of certain codecs (e.g., TTS and STT codecs) used for converting audible speech content into text content and text content into audible speech content. By coordinating the activation and use of these codecs from a central location (e.g., a server), it is possible to improve overall communication between different connected devices in online conferences routed via a server. It is also possible to avoid wasted resources that would otherwise be consumed prematurely by activated codecs for purposes not intended for use and / or before they are needed.

[0036] Now attention will be directed Figure 1 , Figure 1 The illustration depicts a computing environment 100 in which server 110 facilitates electronic communication (e.g., online conferencing and transmission of online conferencing content over different channels) between one or more different client computing systems (120, 160, 170). These client computing systems are also referred to herein as client system A, client system B, and client system C, respectively.

[0037] As illustrated, server 110 includes one or more hardware processors 112 and hardware storage devices 114. Each of the computing systems (120, 160, 170) also includes one or more hardware processors 122 and hardware storage devices 124. Even though computing systems 160 and 170 do not currently show all the same components as those presented in computing system 120, it will be understood that in some embodiments, computing systems 160, 170, and (multiple) third-party systems 180 are each configured to also include all the same components currently shown in computing system 120. Additionally, in some embodiments, such as in a distributed computing environment, server 110 and / or computing system 120 may incorporate one or more additional computing systems, such as computing systems 160, 170, and / or 180.

[0038] During the implementation of the methods disclosed and claimed in this invention, (a plurality of) hardware processors 112, 122 execute computer-executable instructions stored as executable code on (a plurality of) hardware storage devices 114, 124, so as to enable the corresponding server and / or computing system to perform the corresponding actions of the method being implemented.

[0039] During the online meeting, server 110 facilitates / establishes electronic communication between different client computing systems (120, 160, 170) via one or more network connections 130, 132. Further discussion regarding network connections 130, 132 and the different channels and protocols that can be used to transmit electronic communication will refer to... Figure 2 More details are provided below.

[0040] In this brief preview, it is important to note that server 110 can establish electronic communications with multiple different clients to facilitate online communication between multiple different client systems. These electronic communications may include different combinations and types of network connections for each client, including different types of channels and protocols used for processing and transmitting online meeting content. The disclosed embodiments also include switching the type(s) of network connections / channels(s) utilized during online meetings or other communication sessions in response to detected connectivity problems / difficulties, thereby promoting improved reliability in the transmission of online meeting content.

[0041] In some instances, the network connection 130 between server 110 and the first client computing system (e.g., client A) may have the same type / protocol as those used to communicate with the second client computing system (e.g., client B), such as a UDP channel connection used to transmit audio / voice content.

[0042] In some instances, the network connection between server 110 and different client systems (120, 16, 170) can also vary in the type of network connection (130, 132) used. For example, server 110 may use a first type of network connection 132 when communicating with a specific client system (e.g., client C), which is different from those network connections 130 used when communicating with one or more other client systems (e.g., the server may use a UDP channel when communicating with clients A and B, but a TCP channel instead of a UDP channel when communicating with client C).

[0043] When transmitting data to or through server 110, the server uses transport layer 118 to establish the protocol used and manage the correct state and the state of the different channels used to transmit data to or through the server. For example, when a client system joins an online conference, the initial channels and protocols used to transmit the online conference content can be established during session creation.

[0044] Transport layer 118 also tracks the devices of online meeting participants connected to the online meeting, and properly manages the routing of data packets containing online meeting content to the appropriate client devices joining the online meeting. For example, transport layer 118 also manages the connection status of different devices to server 110 and the network communication sessions between different client systems during the online meeting.

[0045] Transport layer 118 is also configured to monitor network connectivity to identify and assess network connectivity issues associated with changes in bandwidth constraints used to transmit online conference content over the network connectivity, and to track packet transmissions to identify, detect, and quantify connectivity issues that could negatively impact the quality of transmission and / or presentation of online conference content that suffers from potential packet loss and / or jitter.

[0046] In some instances, monitoring performed by transport layer 118 includes monitoring for changes in the bit rate of data packets received or transmitted through the server. In other instances, this monitoring includes tracking the identifiers and ordering of data packets transmitted through the server. The monitoring may also include detecting and analyzing notifications from client systems 160, 170, and / or third parties 180 regarding connectivity issues.

[0047] Transport layer 118 also includes logic for generating instructions to activate and / or use a specific codec 150 to process, receive and / or transmit online conference content using the specific codec and / or to process and receive online conference content using different channels and protocols, and to transmit and / or present online conference content in a specific format.

[0048] Transcoding layer 116 is configured to transcode online conference content into a suitable format (e.g., text or audio packets for transmission over UDP or TCP channels) and to transmit it at a specific bit rate to one or more receiving devices (e.g., computing systems 120, 160, 170). In some instances, the server determines the format and bit rate to which data packets are encoded / transcoded based on the detected processing and / or presentation capabilities of each receiving system (e.g., clients A, B, C), and / or based on bandwidth or other network connectivity constraints detected for the communication channels used to communicate with those receiving systems.

[0049] In some instances, transcoding layer 116 includes TTS (text-to-speech) capabilities and / or STT (speech-to-text) capabilities for converting online meeting content from audio to text, and / or for converting online meeting content from text to audio.

[0050] The transport layer 118 and the transcoding layer 116 may each be located entirely at the server 110 and / or at least partially at the client systems 120, 160, and 170 (as transport layer 128 and transcoding layer 126, respectively).

[0051] Computing systems 120, 160, and 170 also include codecs 150 (such as a Text-to-Speech (TTS / STT) codec 152), which are applications and models configured to convert text to speech and speech to text. When instructed, these codecs 150 can be selectively activated to switch and / or use specific codecs to process and transmit online meeting content in a specific format during online meetings, and / or to encode and transmit online meeting content on a specific type of network channel with a specific protocol, and / or to decode and process / present online meeting content received on a specific type of network channel with a specific protocol.

[0052] In some instances, codec 152 in the transmitting client system generating content is specifically configured with speech-to-text functionality to generate text representing speech, and to identify / extract prosodic and speech features from the speech, which can be transmitted as part of the online content along with the generated text. These prosodic and speech features represent one or more of pitch, duration, pauses, loudness, intonation, refraction, style, or multiple other speech attributes / information. Additionally, codec 152 in the receiving client system receiving online content is configured with text-to-speech functionality to generate audio speech corresponding to the text and any associated prosodic and / or speech features received with the text, to present the speech in the desired style. Codec 152 may also be configured to perform language conversion to translate text and / or audio into different languages.

[0053] In some instances, in response to the server's transport layer 118 detecting one or more connectivity issues in electronic communication between the server and client systems, the generation of instructions is automatically triggered at the server. These instructions are used to use or switch the codecs used by the client systems(s), and the one or more connectivity issues negatively affect the quality of the presentation of online meeting content on one or more client systems.

[0054] For example, the detection of connectivity problems may include detecting packet loss or jitter in electronic communications and / or changes in bandwidth availability used for communication between the transmission / reception server system and one of the sending / receiving client systems 120, 160, 170.

[0055] Detecting connectivity issues can also include receiving / detecting user input at the sending or receiving computing system used for online meetings. For example, this user input can be received at the user interface 140 of the client system when a meeting participant visually or aurally detects a problem with the presentation of online meeting content through speakers and / or a display. For instance, if a user detects a defect in the audio being presented through the speakers and / or sees jittery / disjointed video on their display, they can select a connectivity issue button (not shown) displayed within the online meeting interface. In response to the user selecting this button, the client system generates a notification sent to the server informing them of a connectivity problem / difficulty.

[0056] The client system can also independently monitor and detect connectivity problems using the functionality of transport layer 128, as described above, which is similar to the functionality of transport layer 118. By way of example, the client system can automatically detect connectivity problems that negatively affect the transmission and presentation of primary content (e.g., detecting lost packets or a reduction in the bit rate of received packets) and can responsively / automatically notify the server of the detected connectivity problems to trigger the server to automatically generate the instructions(s) described herein.

[0057] After the client system uses its transcoding layer 126 to decode / process the content using a suitable codec 150 (e.g., an application or module), it presents / represents the online conference content at one or more user interfaces 140, where the suitable codec 150 is instructed to process and format the content. The codec 150 can also be used to process / encode the online conference content before transmitting it to the server. For example, the system can use video and audio codecs to encode / decode video and audio data packets. Similarly, other codecs can be used to encode / decode text data packets.

[0058] In some embodiments, codec 150 also includes trained machine learning models, such as trained TTS and / or STT models that are locally accessible within storage device 124 and / or remotely accessible via a third-party system. These trained TTS and / or STT models are trained for one or more languages ​​and styles of speech to convert text to speech and speech to text, respectively. For example, in some instances, the TTS model is trained to generate / present audio from text with one or more distinct / personalized voices, which are presented in one or more predetermined speech styles and / or languages.

[0059] Some codecs 150 can also be configured to convert audio from one language to another in text or audio format. Client systems can store and / or access multiple different models trained to generate / convert audio using different sounds / styles / languages.

[0060] Although not shown, server 110 may also include local storage or remote access to these TTS models via third-party systems, which can be selectively used when transcoding conference content. The system may also be aware of available codecs and models that can be used by different client systems and, when determined to be appropriate (e.g., in response to client / user preferences and constraints, and in response to detected network connectivity issues), generate suitable instructions to the client system to activate, use, or convert them for use.

[0061] In some instances, instructions received from the server specify the particular codec(s) to be used and / or the method for processing the online meeting content, specifically identifying the TTS(s) models(s) and / or personalized voices / speaking styles / languages(s) used to process, transmit, receive, and / or present the online meeting content. The client system may accept and comply with the instructions. Alternatively, due to user input and / or system constraints, the client system may reject one or more of the received instructions, as will be described in more detail below.

[0062] When the online conference content is first received and processed by a sending computing system (e.g., system 120), the content is encoded into a specific format and transmitted to one or more receiving devices via server 110. The content may undergo further transcoding (format or bitrate change) on the server and is ultimately transmitted from the server to the receiving devices (e.g., computing systems 160, 170). For example, by way of example, when the online conference content is audio or video, the client system can encode the online conference content into a format suitable for transmission with the server over a UDP network channel. When the content is text, the online conference content can be encoded into a format suitable for transmission with the server over a TCP network channel. The client system can initially encode the online conference content according to instructions received from the server, and these instructions can be changed throughout the online conference.

[0063] Once encoded, data packets are transmitted via server 110 to receiving devices (e.g., client systems 160, 170) using the same type of network channel / protocol (e.g., UDP or TCP channel type) or a different type of network channel / protocol. This same type of network channel / protocol is also used to transmit data packets from sending systems (e.g., client 120) to server 110. For example, in some instances, particularly when data packets undergo additional transcoding at server 110, the server may use a different type of channel / protocol to receive and send data packets between different client systems (e.g., a UDP channel to receive audio packets transcoded to text at the server, and then sending those audio packets to the receiving system via a TCP channel).

[0064] In some instances, a server can transmit received online meeting content to one or more clients using the same format and bitrate without further transcoding. For example, a server can transmit received audio content to the receiving device without transcoding the meeting content from audio format to text format.

[0065] Even without converting the meeting content from audio to text at the server, the server can still transcode the content. For example, due to detected network connectivity issues with the receiving device (e.g., poor connection, bandwidth limitations, or other connectivity problems preventing the receiving device from receiving data packets at a relatively high UDP bit rate), and / or due to presentation constraints of the receiving device, content received on a first-type network channel / protocol (e.g., UDP) can be transcoded into a different format for more reliable transmission on a less lossy channel (e.g., TCP).

[0066] The server can determine client configuration and settings by requesting and retrieving information from the client and / or by receiving spontaneous information from the client, which identifies client configuration, constraints, and / or user settings. In a non-limiting example, if the receiving device has online meeting parameters that are manually configured to present text instead of audio due to user selection or automatically due to detected device limitations (e.g., a faulty speaker or an disabled speaker interface), the server can detect these online meeting parameters by requesting system configuration settings from the client before transmitting the online meeting content to the user. Based on this information, the server can then select an appropriate format / channel for transmitting the online meeting content, as well as an appropriate format for transcoding and / or transmitting the online meeting content.

[0067] During the meeting, changes in system configuration and settings can be detected in real time by the server and / or client systems to modify the format / channel used to transmit online meeting content, and to trigger instructions from the server to activate, use, or switch the codec / channel used to process and transmit / receive meeting content. When changes are made, the server tracks those changes to ensure it knows when to generate and issue new instructions to the different client systems participating in the online meeting.

[0068] In some embodiments, the server transcodes content received from a first client in multiple different ways to use different formats and / or bitrates for each of multiple different receiving devices that transmit the same content in different formats over the same and / or different types of network channels / protocols (e.g., audio format, text format, first bitrate, second bitrate, UDP format, TCP format, or any combination of other formats). The server also generates and sends corresponding instructions to different client systems to use, activate, and switch the codecs being used to process the data accordingly.

[0069] Then, when receiving content at different client systems, regardless of the (multiple) formats in which the content is received, it will be presented in one or more different formats at different receiving systems, which activate and use appropriate codecs according to instructions. For example, after decoding and selectively converting the content according to instructions received from the server, different receiving systems can, according to instructions, apply or omit additional processing by the client system codecs to present the same content in one or more different formats at different receiving systems (e.g., by presenting the content as audio by all systems, by presenting the content as text by all systems, by presenting the content as both text and audio by all systems, by presenting the content as audio by some systems and as text by other systems, by presenting the content as either audio or text by some systems and as both audio and text by other systems, etc.).

[0070] As server 110 establishes and monitors network connections with different client systems as part of an online conference session, transport layer 118 tracks which channels, protocols, and data formats are used to transmit online conference content with each computing system. As discussed throughout, the server also determines when to generate instructions for clients to activate, switch, and / or use different codecs 150 for receiving, transmitting, presenting, or otherwise processing online conference content by utilizing specific formats, protocols, and network channels. Then, when changes are made at the client level and / or to the protocols / channels / formats used for transmitting online conference content, those changes are also tracked by transport layer 118 at the server level during the online conference.

[0071] However, it's important to note that not all instructions result in the specified changes. For example, when presenting online meeting content, a server instruction might command the receiving system to activate a TTS encoder to convert and present text data as audio. In response, the receiving system can comply and activate the appropriate codec to receive the online meeting content over the TCP channel, and process the text / convert the text to audio for presentation through speakers. Alternatively, if converting text to audio using the STT codec 152 fails, the receiving system can disregard the instruction by simply activating the appropriate codec for receiving / processing data over the TCP channel. The client system can choose to perform this step by selecting to present the text as displayed characters on the screen instead of audio through speakers.

[0072] In some instances, the receiving device may choose to disregard part of the server's instructions based on user input received at the receiving device and / or automatically based on constraints at the client system (e.g., determining that there is no suitable audio speaker or the system does not have access or license approval to use a particular STT codec 152 that is required to convert text into a specified language, sound profile, style, or other audio format specified to be used when presenting content).

[0073] When presenting content, the client system can also present online meeting content in a requested / specified format, as well as in a non-specified format. For example, the system can present content including audio in audio format at one or more speakers, and simultaneously in text format on one or more displays.

[0074] Throughout the online meeting, different client systems may choose to present the received online meeting audio content (whether received as audio or text) in the same and / or converted audio and / or text format, regardless of and / or based on instructions received from server system 110.

[0075] As previously mentioned, server 110 can establish different combinations of network connections with different client systems. Some non-limiting examples of different types of network connections will now be referenced. Figure 2 describe.

[0076] Figure 2 Two different types of network connections are illustrated in particular. The first network connection is a connectionless protocol channel (such as UDP), which is lossy and suffers from jitter, especially in challenging network conditions that lead to packet loss and delay. This channel is configured to transmit audio / video data packets between (multiple) client computing systems and servers at a bit rate of approximately 10-60 Kbit / sec, or faster.

[0077] The second network connection is a connection-based protocol channel (such as TCP), which is slower than the first channel but generally more reliable than the first UDP channel. In some instances, the TCP channel is lossless or jitter-free, or at least has less loss and is less sensitive to jitter than the first channel. In some instances, the second channel is used to transmit text data packets. Note that when audio data is converted to text or formatted into different languages, it is possible to transmit the same content (e.g., voice) in different formats over different channels according to different protocols.

[0078] The disclosed and claimed embodiments may use both of the aforementioned network connections and / or any other network connection to transmit online conference content between different client computing systems via a server. As described herein, the server may also switch the channel and network connection in use during a single continuous and / or uninterrupted online conference session involving one or more online conference participant computing systems.

[0079] When switching content formats or network connections, the switch may require the server to transcode the online meeting content into one or more different formats. The server will generate and send corresponding instructions to different client systems to activate, use, switch, or otherwise use different / specific codecs. The instructions can specifically identify the codec to be used.

[0080] Alternatively, the instruction can be an inference instruction that specifies a new network channel / protocol for transmitting / receiving online conference content and / or for formatting online conference content in a particular manner, without specifically identifying the codec to be used. Such an inference instruction leaves the client system with the autonomy to select and use appropriate (available and relevant) codecs to process online conference content in the specified format and using the specified channel / protocol.

[0081] Attention will now be directed to Figures 3 to 6 , Figures 3 to 6 The diagram illustrates a flowchart of actions associated with the methods used to achieve the disclosed functionality. Figure 3 The diagram illustrates a flowchart of actions performed from the perspective of a media generation client system that generates online meeting content via a server and sends it to another system. Figure 4 The diagram illustrates a flowchart of actions performed by the receiving media consumption / presentation client system.

[0082] Figure 5 A flowchart is shown, including actions performed by an intermediate server. Furthermore, Figure 6 Includes Figures 3 to 5 The composite flowchart of the actions referenced in the text illustrates the ordering, relevance, and interrelationships between different server and client systems.

[0083] Regarding the following Figures 3 to 6 The discussion will help us understand that although a particular order of processes is illustrated in a flowchart, the specific order is not specifically required unless otherwise stated or required, because actions depend on another action that is performed before the action is performed.

[0084] Now, for reference Figure 3It should be noted that the illustrated flowchart 300 includes a media generation client (such as computing system 120) that establishes electronic communication with a server (e.g., server 110) using a first channel of a first protocol (e.g., a UDP channel) (action 310). The client system also activates a first set of suitable codecs for communication over the first channel of the first protocol (action 320). For example, this action may be performed in response to or independently of instructions from the server when establishing an online conference session with the server.

[0085] Then, during the online meeting, the system detects user input, such as speaker audio input received at the system's microphone (Action 330). The system then encodes and transmits the audio using a first protocol on the first channel (Action 340). In some instances, the client system may also convert the audio to different languages ​​and / or encode it into an audio format determined to be compatible with the capabilities of the intended receiving device based on instructions received from the server, so that the server does not have to perform this function.

[0086] In some instances, the client can independently detect connectivity issues based on monitoring network connectivity with the server and / or based on user input (Action 395). This can generate instructions at the client to use / activate different codecs for encoding audio. It is also possible to notify the client of network connectivity issues by receiving instructions from the server to activate and encode audio using one or more codecs, and / or initiate different encodings of audio to different formats (Action 350).

[0087] Based on instructions generated in response to detected connectivity issues, the client converts (Action 370) the same or newly detected audio (Action 360) into a suitable format. The system also establishes / utilizes electronic communication with the server to encode and / or transmit audio content (Action 380), which includes a second channel using a second protocol. For example, this could include encoding audio into text for transmission to the server over the second channel. The codec(s) can also extract / generate speech attribute information, such as prosody and speech features associated with the audio / speech that can be transmitted as part of the transmitted online content along with the text, and these prosody and speech features representing one or more of the following speech attributes / information: pitch, duration, pauses, loudness, intonation, refraction, style, or other speech attributes / information associated with the audio converted to text. In this regard, new conference content received at the client system can then be converted into both text and corresponding speech attribute information associated with the audio converted to text, which is useful for converting the text back to audio of the same / similar style. In some instances, the second channel is a TCP channel and the first channel is a UDP channel. See reference... Figure 2Other types of channels can also be used, as described.

[0088] Figure 4 The diagram illustrates a flowchart 400 reflecting how a receiving computing system receives and processes online meeting content and how it responds to instructions from server 110 to change how it processes the online meeting content in response to dynamic, changing, and / or challenging network conditions. The described receiving computing system may include computing systems 160, 170, or a third-party system 180, each of which may also be configured as computing system 120.

[0089] As shown in the figure, the receiving computing system establishes electronic communication with a server (e.g., server 110) using a first channel of a first protocol (e.g., a UDP channel) (action 410). The client system also activates a first set of suitable codecs for communication using the first channel of the first protocol (action 420). For example, this action may be performed in response to or independently of instructions from the server when establishing an online conference session with the server.

[0090] Then, during the online meeting, the system receives / detects encoded audio (action 430), such as data packets containing audible speech, which are decoded and presented as audio using appropriate / corresponding codecs (multiple) at the receiving client system and a speaker (or other output device) (action 440). These codecs may include any of the codecs 150 described above, and these codecs may be used to play the audio and / or translate the audio into different languages ​​or styles. In some instances, even if not explicitly shown, the system may also use codecs to convert the audio into text captions that are displayed on the screen simultaneously with the presented audio during the online meeting (action 440).

[0091] At certain points during the meeting, the receiving system can independently detect connectivity issues based on monitoring network connectivity with the server and / or based on user input (Action 395). For example, a user at the receiving system can provide input indicating or reflecting cross-network connectivity issues or inconsistencies in media presentation associated with connectivity problems / difficulties.

[0092] This input allows the receiving system to generate instructions (action 480) for using / activating different codecs used to decode / present audio, such as in a text format that helps the user more clearly explain what is being said, especially when a trained STT model can be used to intelligently generate or correct missing items in a portion of the encoded audio that are not received in complete / correct format. The receiving system can also notify the server of such connectivity issues / difficulties when they are detected at its location.

[0093] The client network can also be individually notified of connectivity issues (Action 490) by receiving instructions from the server to activate and use one or more codecs in order to receive the same or corresponding media content (such as text content transcoded / converted from audio to text before being transmitted to the receiving system) via different channels. These instructions may be the result of network connectivity issues detected by the server independently and / or in response to input received from the sending or receiving computing system.

[0094] Finally, the receiving system establishes communication with the server on a second channel using a different protocol than the first channel (action 450). This action can be performed before or in response to the instruction received in action 480. In some embodiments, the second channel is a TCP channel, and the first channel is a UDP channel.

[0095] The receiving system also receives encoded text and, optionally, associated voice attribute information via a second channel using a different protocol than the first channel (Action 460). This text content may be the same online conference content (e.g., the same speech / words) included in the audio content referenced in Action 430, but only formatted differently. Alternatively, the newly received content may be entirely different online media content subsequently received and generated by the system, which generates and initially transmits online conference content relative to the previously received audio content referenced in Action 430.

[0096] The receiving system also utilizes one or more suitable codecs to decode and present the new text context (Action 470). In some instances, the system uses codec(s) to decode and present the text in text format on a display at the receiving system (Action 472). In some instances, the system will utilize codec(s), additionally or alternatively, to convert the text into audio and present the audio in a specific language and style on one or more speakers at the receiving system (474). In some instances, codec(s) may also use associated speech attribute information transmitted with the text to present the audio in the same prosody, speech style, and language as originally presented at the transmitting client system. Alternatively, codec(s) may present the audio in a different style, language, or expression than originally presented at the transmitting client system.

[0097] As previously documented, the reception of content by the receiving system through different channels, and in particular the processing and presentation of content using (multiple) specific codecs, can dynamically change (explicitly or implicitly) during online meetings in response to instructions generated by and received from the server, and instructions are automatically triggered by the server system in response to connectivity issues affecting the quality of online meeting transmission and / or presentation.

[0098] Attention is now directed to Figure 5 , Figure 5 The flowchart 500 is illustrated, which shows the actions from the perspective of a server system (such as server 110) communicating with different client systems (e.g., client systems A, B & C).

[0099] As noted, the server establishes electronic communication with one or more clients over one or more first channels using a first protocol (e.g., UDP) (Action 510). The server may have several UDP channels established using each of multiple different client systems; for example, a first client may generate content to be presented by another client. During an online meeting, each of the client systems may independently generate content to be presented by other connected client systems. Each of the client systems may also be configured to present content generated by other client systems.

[0100] The server acts as an intermediary, receiving, processing, and transmitting client data (which may include video, audio, and / or text data) generated during online meetings to other clients. In some instances, client data, including audio, is received and / or transmitted to one or more transmissions (action 520) on one or more first channels of a first type (e.g., UDP) to different participants' computing systems. This may include transcoding audio into different formats, such as different audio formats (e.g., language, bitrate, etc.). This may also include modifying audio into different formats, such as text formats. The action may also include changing the format of text content to different formats (e.g., different fonts, languages, bitrates, etc.) and converting text to audio.

[0101] The server also establishes electronic communication with the same and / or different clients via one or more second channels having different protocols than the first channels (Action 550). Based on determining which channels and formats will result in the fastest transmission of data to each of the different clients with optimal reliability or at least within a predetermined threshold reliability or QoS (Quality of Service) associated with transmission bit rate and reliability (e.g., no data loss and no jitter), the server uses and selects the best / suitable (multiple) channels / data formatting for transcoding data and transmitting data to the different clients. The server can also determine the channel / data formatting to use based on explicit user input received from the user at the online conferencing interface during the conference.

[0102] During the meeting, the server monitors network connections with one or more clients on one or more channels (Action 530). During this monitoring, the server can detect network connectivity problems / difficulties (Action 532). These connectivity problems may be associated with data loss or jitter and can be detected by identifying bandwidth limitations, server constraints, network transmission errors, packet loss, persistent bit rate reductions, etc.

[0103] Connectivity issues can also be identified based on receiving a notification of the problem from another source (Action 534), such as a network problem notification from a third-party source, user input from one or more places in the online meeting participant system, notifications generated by the online meeting participant system independently of user input, and so on.

[0104] The detection or identification of network connectivity problems can automatically trigger the generation and transmission of instructions for (multiple) client systems to activate, switch, use, or otherwise utilize different codecs to encode, decode, present, transmit, receive, or otherwise process online conference content and / or utilize one or more different protocols / channels to transmit or receive online conference content (Action 540), as described throughout this document.

[0105] When a server establishes a new connection with a client system using different protocols (Action 550), the instruction can be implicit. Additionally or alternatively, the instruction can explicitly command (multiple) client systems to use which (multiple) codecs to process the online meeting content. The instruction can also command how the content is presented.

[0106] The server may selectively determine whether a network connectivity problem warrants modification before generating instructions and / or before establishing (multiple) different communication channels. For example, the server may determine (individually or collectively) whether the network connectivity problem is of a specific type (e.g., packet loss, out-of-order packets, user-defined quality, bandwidth changes, no response, etc.) and / or quantity (e.g., a certain percentage of data packets lost, a certain percentage of out-of-order data packets, a percentage change in transmission bit rate, duration of no response, etc.), and / or whether the network connectivity problem persists for a specific duration or frequency, and / or whether the network connectivity problem has been verified before generating / transmitting the aforementioned instructions and / or before establishing a new / secondary communication channel with (multiple) client systems.

[0107] By way of example, if intermittent interruptions in bandwidth availability on the first channel (e.g., UDP channel) periodically slow down transmission, but are insufficient to cause data loss or jitter due to adequate buffering of data at the server / client system, the server can determine to suppress the generation / transmission instructions so that the client system switches to the channel / protocol / codec used to transmit / process online conference content in a different format (e.g., converting audio to text) for transmission / processing on a different channel (e.g., TCP channel).

[0108] During an online meeting, after changes have been made to the content formatting and the channels(s) used to transmit the content, the server can also generate additional instructions to switch back to a previous configuration (e.g., protocol / formatting) or a different configuration. This different configuration is determined to be more efficient, reliable, or faster than the current configuration for processing and transmitting / receiving online meeting content due to newly detected network connectivity conditions. The new protocol / formatting configuration used in subsequent switches and on which the new instructions are based can be the same configuration previously used during the same online meeting, or alternatively, the new configuration can be a completely different configuration of the protocol / formatting not previously used in the same online meeting.

[0109] Due to newly detected network connectivity issues, additional instructions can be generated to improve transmission rates and / or reliability, and these instructions may include improved network conditions. In this way, the current system can be used to dynamically adjust the configuration used during online meetings to maximize the efficiency and reliable transmission of online meeting content.

[0110] Attention will now be directed to Figure 6 , Figure 6 Including about Figures 3 to 5 The same actions described above. However, in this diagram, the composite flowcharts (300, 400, 500) illustrate a non-limiting example of how certain actions can be related through the specific dependencies and ordering of these actions, which is relevant to the reference. Figure 3-5 The flowcharts and embodiments described may not be obvious or required. Despite this potential difference in dependencies / order, Figure 6 The actions shown will not be repeated or described again, as this is not necessary. In particular, the actions shown in flowchart 600, as will be understood, are combined with those previously referenced. Figure 3-5 The actions described, as well as all the functions described throughout the remainder of this disclosure.

[0111] It will also be understood that, regarding the foregoing, the methods and corresponding actions cited can be found, for example, in reference to Figure 1 and Figure 2 The described server and client computing systems are implemented or practiced in computer systems. These systems include one or more processors and computer-readable hardware storage / media such as computer memory and other hardware storage devices. Specifically, computer memory may store computer-executable instructions that, when executed by one or more processors, cause various functions to be performed, such as the actions enumerated in the embodiments.

[0112] Embodiments of the present invention may include or utilize dedicated or general-purpose computers including computer hardware, as discussed in more detail below. Embodiments within the scope of the present invention also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media accessible by a general-purpose or dedicated computer system. Computer-readable media storing computer-executable instructions are physical hardware storage devices excluding carrier waves or signals. Computer-readable media carrying computer-executable instructions are transmission media. Therefore, by way of example and not limitation, embodiments of the present invention may include at least two distinct types of computer-readable media: physical computer-readable storage media and transmission computer-readable media.

[0113] Physical / hardware computer-readable storage media / devices include RAM, ROM, EEPROM, CD-ROM or other optical disc storage (such as CD, DVD, etc.), disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer.

[0114] Furthermore, upon arrival at various computer system components, program code in the form of computer-executable instructions or data structures can be automatically transferred from a transport computer-readable medium to a physical computer-readable storage medium (and vice versa). For example, computer-executable instructions or data structures received via a network or data link can be cached in RAM within a network interface module (e.g., a "NIC") and then ultimately transferred to the computer system RAM and / or a less volatile computer-readable physical storage medium at the computer system. Therefore, computer-readable physical storage media can be included in computer system components that also (or even primarily) utilize the transport medium.

[0115] Computer-executable instructions include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a particular function or set of functions. Computer-executable instructions can be, for example, binary code, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or actions described above. Rather, the described features and actions are disclosed as exemplary forms for implementing the claims.

[0116] Those skilled in the art will understand that this invention can be practiced in network computing environments with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframes, mobile phones, PDAs, pagers, routers, switches, etc. This invention can also be implemented in distributed system environments, where both local and remote computer systems connected via a network (via a hardwired data link, a wireless data link, or a combination of hardwired and wireless data links) perform tasks. In a distributed system environment, program modules can reside on both local and remote memory storage devices.

[0117] Alternatively or additionally, the functions described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), etc.

[0118] The invention may be embodied in other specific forms without departing from its essential characteristics. The described embodiments are to be considered illustrative rather than restrictive in all respects. Therefore, the scope of the invention is indicated by the appended claims rather than by the foregoing description. All modifications within the meaning and scope of equivalents of the claims are included within their scope.

Claims

1. A method implemented by a server system for managing communication between online meeting participants and mitigating connectivity issues associated with one or more meeting participant devices to improve the reliability of the communication despite the presence of the connectivity issues, the method comprising: Establish electronic communications with multiple different clients to facilitate online conference communications between the multiple different client systems via the electronic communications and the server system, the multiple different client systems including a transmitting client system and a receiving client system, the electronic communications between the server system and the transmitting client system including a first channel for transmitting content associated with the online conference using a first protocol; The transmission client system receives first online conference content on the first channel, the first online conference content including audio data obtained by the transmission client system, and the first online conference content is encoded and transmitted to the server system by the transmission client system using the first protocol and in an audio data format; The first online conference content received from the transmitting client system is transmitted to the receiving client system on the first channel; Upon receiving the content of the first online meeting, an instruction is generated and sent to the transmission client system to activate one or more codecs at the transmission client system. This initiates the conversion and encoding of audio obtained by the transmission client system from the online meeting into text data and associated speech attribute information. The codecs have speech-to-text functionality. The method further includes: The system receives prosodic and speech features extracted from the audio, wherein the prosodic and speech features include one or more of the following: pitch, duration, pause, loudness, tone, foldback, or style; and The prosody and speech features are transmitted to the receiving client system, wherein the prosody and speech features are used by the receiving client system to generate audio speech corresponding to the text data in a desired style.

2. The method of claim 1, wherein the method further comprises: New online conference content is received from the transmission client on a second channel as text data having the associated voice attribute information. The second channel has a lower bit rate and a different protocol than the first channel. The new online conference content is received after the instruction is sent, and the text data has been converted from audio to text data by the codec at the transmission client system in response to the instruction.

3. The method according to claim 1, wherein the method further comprises: At the server system, the content of the first online meeting is transcoded into text.

4. The method according to claim 1, wherein the method further comprises: The first protocol is used to transmit the first online meeting content received from the transmitting client to the receiving client system, without transcoding the first online meeting content from the audio data format to the text data format.

5. The method according to claim 1, wherein the method further comprises: After first transcoding the first online conference content from the audio data format to the text data format, a second protocol with less loss than the first protocol is used to transmit the first online conference content received from the transmitting client to the receiving client system.

6. The method according to claim 1, wherein the method further comprises: Generate and send different instructions to the receiving client system to activate one or more codecs at the receiving client system, so as to initiate the decoding of online meeting content using a codec with text-to-speech functionality.

7. The method of claim 1, wherein the generation of the instruction is automatically triggered in response to the server system detecting one or more connectivity problems in the electronic communication between the server system and the transmitting client system and / or the receiving client system, the one or more connectivity problems negatively affecting the quality of the content of the online meeting presented at the receiving client system.

8. The method of claim 7, wherein the detected connectivity problem includes packet loss.

9. The method of claim 7, wherein the detected connectivity problem includes jitter.

10. The method of claim 7, wherein the detection of the connectivity problem includes receiving input from a user participating in the online meeting.

11. The method of claim 10, wherein the user input is received from the receiving client system.

12. The method of claim 7, wherein the detection of the connectivity problem comprises: Detect changes in bandwidth availability used for transmitting communication between the server system and the client system.

13. The method of claim 7, wherein the detection of the connectivity problem comprises: Detect changes in bandwidth availability used for transmitting communication between the server system and the receiving client system.

14. A method implemented by a receiving client system for processing online meeting content during an online meeting between online meeting participants and mitigating connectivity issues associated with one or more meeting participant devices to improve the reliability of communication despite the presence of the connectivity issues, the method comprising: The receiving client system establishes electronic communication with the server system, the server system being configured to facilitate the transmission of online conference content generated by the transmitting client system to one or more other client systems, including the receiving client system, during the online conference, the electronic communication between the server system and the receiving client system including a specific channel for transmitting content associated with the online conference using a specific protocol; The first online conference content is received and decoded on the specific channel using the first protocol, the first online conference content being received in audio data format; After receiving the first online meeting content, an instruction is received from the server system to activate one or more codecs at the receiving client system with text-to-speech functionality that can decode the online meeting content into text format. In response to the instruction, activate the one or more codecs; Establish new electronic communication with the server system on different channels, the different channels using different protocols compared to the specific protocol; On the different channels using the different protocols, new online conference content is received and decoded using one or more codecs having the text-to-speech functionality, the new online conference content being received in a text format having speech attribute information associated with the text; as well as The new online meeting content is presented at the receiving client system, and the new online meeting content is decoded and presented using one or more codecs. The method further includes: The server system receives prosodic and speech features extracted from the audio of the first online meeting content, wherein the prosodic and speech features include one or more of the following: pitch, duration, pause, loudness, tone, retraction, or style; Based on the prosody and speech features, generate audio speech corresponding to the text; and The audio speech is presented in the intended style.

15. The method of claim 14, wherein the presentation of the new online meeting content comprises: The new online meeting content is presented in text format on the display of the receiving client system.

16. The method of claim 14, wherein the presentation of the new online meeting content comprises: After the new online meeting content is converted from the text format to the audio format using the one or more codecs based at least in part on the voice attribute information received along with the text, the new online meeting content is presented in audio format using the speaker of the receiving client system.

17. The method of claim 14, wherein the method further comprises: Prior to receiving the instruction, the receiving client system receives user input, which identifies difficulties or problems associated with presenting the first online meeting content at the client system, and a notification to transmit the user input to the server system.

18. The method of claim 14, wherein the method further comprises: Before receiving the instruction, the system automatically detects connectivity issues that negatively affect the transmission or presentation of the first online meeting content, and notifies the server system of the detected connectivity issues, thereby triggering the server system to generate the instruction.

19. A method implemented by a transmission client system for generating and transmitting online meeting content during an online meeting between online meeting participants, and for mitigating connectivity issues associated with one or more meeting participant devices to improve the reliability of communication despite the presence of the connectivity issues, the method comprising: Establish electronic communication with a server system configured to facilitate the transmission of online meeting content generated by the transmitting client system to one or more other client systems during the online meeting, the electronic communication between the server system and the transmitting client system including a first channel for transmitting content associated with the online meeting using a first protocol; Generate first online meeting content, and transmit the first online meeting content to the server system in audio data format on the first channel using the first protocol; After transmitting the first online meeting content, an instruction is received from the server system to activate one or more codecs at the transmitting client system to initiate the transcoding of the new online meeting content from the audio format to the text format, wherein the one or more codecs have speech-to-text functionality. In response to the instruction, activate the one or more codecs; Identify new online meeting content including audio, and use one or more codecs to convert the new online meeting content into text and corresponding voice attribute information; as well as The new online meeting content is transmitted to the server system via a second channel using a second protocol, which is different from the first protocol. The method further includes: Prosodic and speech features are extracted from the audio of the online meeting content, wherein the prosodic and speech features include one or more of the following: pitch, duration, pause, loudness, tone, retraction, or style; and The prosody and speech features are sent to the server system, wherein the prosody and speech features are used by the receiving client system to generate audio speech corresponding to the text in a desired style.

20. The method of claim 19, wherein the first protocol is User Datagram Protocol (UDP) and wherein the second protocol is Transmission Control Protocol (TCP).

Images