Media adaptation for lost content recovery in real-time online conferences

Abstract

An apparatus and method for recovering lost real-time online conference content is a computer-implemented method. During a real-time online conference, real-time conference data including conference content in a data format is transmitted between participant devices. During transmissions between these devices, some data may be lost. And when the lost real-time conference data is lost, acquiring real-time recovery data in different data formats. The conference content in the lost real-time conference data is replaced by the real-time recovery data. And rendering the recovery data on the participant device with the lost data to replace the real-time conference content in the lost data.

Description

Technical Field

[0001] The present disclosure generally relates to improving the quality of audio / visual interactions in real-time conferences conducted over a communication network. Background Art

[0002] The use of real-time video conferencing applications has increased significantly in recent years. Typical video conferencing falls into two categories: group video conferencing with two or more participants, who can all see and communicate with each other in real time; and online presentations, in which one or more presenters present information to a large group of attendees using audio, visuals, and text. Both types rely on fast and reliable network connections for efficient meetings and presentations, but quality can be affected when the network bandwidth between the presenter and attendees fluctuates or is limited.

[0003] Typically, when a network problem occurs, participants may lose the audio portion or the video portion or both of the conference. During a real-time conference, there is little that participants or conference applications can do to ensure that participants will not miss the portion of the conference that was interrupted by the network problem.

[0004] Quality of experience (QoE) is a measure of the customer's experience of a service and is one of the most commonly used service metrics for measuring video delivery performance. QoE describes the performance of a service from the user's or viewer's perspective. Typically, video content is the most bandwidth-intensive part of a speech-based presentation. When using holographic, 3D, or volumetric video conferencing services, video quality becomes even more bandwidth-intensive. Summary of the Invention

[0005] A general aspect includes a computer-implemented method for recovering lost real-time online conference content. The computer-implemented method includes: transmitting real-time conference data including conference content in a certain data format to a participant device during a real-time online conference and receiving real-time conference data including conference content in a certain data format from a participant device; determining that the real-time conference data transmitted sequentially between devices during the real-time online conference has been lost. The method also includes: receiving real-time recovery data in a different data format, the real-time recovery data containing real-time online conference content that replaces the conference content in the lost real-time conference data. The method also includes: rendering the real-time recovery data to replace the real-time conference content - replacing what was lost. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0006] Implementations may include any of the above-described implementations of the computer-implemented method, wherein the conference content is in an audio / visual format and the different data formats include non-audio / visual formats. Implementations may include any of the above-described implementations of the computer-implemented method, wherein rendering may include rendering the real-time recovery data at the same time as rendering the real-time conference data received after the lost real-time conference data is lost. Implementations may include any of the above-described implementations of the computer-implemented method, wherein the real-time recovery data may include a text transcription of the audio / visual data and rendering may include rendering the text transcription of the audio / visual data in the real-time recovery data at the same time as rendering the real-time conference data received after the lost real-time conference data is lost. Implementations may include any of the above-described implementations of the computer-implemented method, wherein the conference content is in an audio / visual format and the different data formats may include a playback rendering optimized format. The playback rendering optimized content may include audio / visual content having a subset of both the audio data and the visual data in the lost real-time conference data. Implementations may include any of the above-described implementations of the computer-implemented method, wherein the playback rendering optimized content may include audio / visual content having a subset of audio data less than all of the audio data in the lost real-time conference data. Implementations may include any of the above-described implementations of the computer-implemented method, wherein the playback rendering optimized content may include audio / visual content with empty portions of the audio / visual content removed. Implementations may include any of the above-described implementations of the computer-implemented method, wherein the real-time recovery data is received from a cache on a network device in the network environment. Implementations of the described technology may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0007] Another aspect includes a computer-implemented method for recovering real-time conference content lost during an online presentation. The computer-implemented method further includes: transmitting real-time conference data including real-time conference content in a certain data format to a participant device during the real-time online conference and receiving real-time conference data including real-time conference content in a certain data format from the participant device; generating real-time content recovery data in a different data format, the real-time recovery data replacing the conference content in the real-time conference data. The method further includes: forwarding the real-time recovery data to at least one participant device to replace the real-time conference content determined to be lost in communication with the host device, the real-time recovery data including content that replaces the lost content.

[0008] Implementations may include any of the above-described implementations of the computer-implemented method, wherein the conference content is in an audio / visual format and the different data format is a non-audio / visual format. Implementations may include any of the above-described implementations of the computer-implemented method, wherein the conference content is in an audio / visual format and the different data format includes a playback rendering optimized format. Implementations may include any of the above-described implementations of the computer-implemented method, wherein generating may include creating playback rendering optimized content having the following audio / visual content, wherein the audio / visual content is a subset of both the audio data and the visual data in the lost real-time conference data. Implementations may include any of the above-described implementations of the computer-implemented method, wherein generating may include generating playback rendering optimized content having only audio data, and which is a subset of audio / visual content that is less than all audio / visual content in the lost real-time conference data. Implementations may include any of the above-described implementations of the computer-implemented method, wherein the playback rendering optimized content may include audio / visual content with empty portions of the audio / visual content removed. Implementations may include any of the above implementations of the computer-implemented method, wherein generating may include converting conference content in conference data lost in communications with the host device into a transcription of the lost conference data. Implementations may include any of the above implementations of the computer-implemented method, wherein generating may include generating a text transcription of the audio / visual data that is forwarded for rendering at the same time as real-time conference data received after rendering of the lost real-time conference data is lost.

[0009] A general aspect includes a processing system in a network. The processing system includes: a processor-readable storage medium; a processor device including a first non-volatile memory storage device may include instructions; and one or more processors in communication with the memory. The one or more processors execute the instructions to perform the following operations: routing real-time conference data transmitted between a source device and at least one participant processing device during a real-time online conference, the real-time conference data including conference content in a certain data format to the participant device during the real-time online conference and conference content in a certain data format from the participant device; generating real-time recovery data in a different data format, the real-time recovery data replacing the real-time conference content in the real-time conference data; storing the real-time recovery data; and forwarding the real-time recovery data to at least one participant device to replace the real-time conference content determined to be lost in communication with the at least one participant device, the real-time recovery data including content that replaces the lost content. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0010] Implementations may include one or more of the above-described processing systems, wherein the conference content is in an audio / visual format and the different data formats include non-audio / visual formats. Implementations may include one or more of the above-described processing systems, wherein one or more first processors execute instructions to generate by converting conference content in conference data lost in communication with a host device into a transcription of the lost conference data. Implementations may include one or more of the above-described processing systems, wherein the real-time recovery data may include a text transcription of the audio / visual data, which is forwarded for rendering at the same time as the real-time conference data received after the lost real-time conference data is lost. Implementations may include one or more of the above-described processing systems, wherein the conference content is in an audio / visual format and the different data formats include a playback rendering optimized format. Implementations may include one or more of the above-described processing systems, wherein the playback rendering optimized content may include audio / visual content having a subset of both the audio data and the visual data in the lost real-time conference data. Implementations may include one or more of the above-described processing systems, wherein the playback rendering optimized content may include audio / visual content having a subset of audio data that is less than all of the audio data in the lost real-time conference data.

[0011] On the other hand, a user device is included. The user device also includes a storage medium, which includes computer instructions. The device includes: one or more processors, which are coupled to communicate with the storage medium, wherein the one or more processors execute the instructions to cause the system to perform the following operations: transmitting real-time conference data including real-time conference content in a certain data format to a participant device during a real-time online conference and receiving real-time conference data including real-time conference content in a certain data format from a participant device; determining that the real-time conference data transmitted between devices during the real-time online conference has been lost; receiving real-time recovery data in a different data format, the real-time recovery data replacing the real-time conference content in the lost real-time conference data; and rendering the real-time recovery data to replace the real-time conference content in the lost real-time conference data. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of the method.

[0012] Implementations may include a user device, wherein the conference content is in an audio / visual format and the different data formats include a non-audio / visual format. Implementations may include a user device, wherein the conference content is in an audio / visual format and the different data formats may include a playback rendering optimized format. Implementations may include a user device, wherein the playback rendering optimized content may include audio / visual content having a subset of both the audio data and the visual data in the lost real-time conference data. Implementations may include a user device, wherein the playback rendering optimized content may include audio / visual content with empty portions of the audio / visual content removed. Implementations may include a user device, wherein the playback rendering optimized content may include audio / visual content having a subset of less than all of the audio data in the lost real-time conference data. Implementations may include a user device, wherein rendering may include forwarding the real-time recovery data to be rendered at the same time as rendering the real-time conference data received after the lost real-time conference data is lost. The real-time recovery data may include a text transcription of the audio / visual data.

[0013] This summary is provided to introduce some concepts that are further described below in a simplified form in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. The claimed subject matter is not limited to implementations that solve any or all of the disadvantages mentioned in the background. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The various aspects of the present disclosure are illustrated by way of example and not limitation in the accompanying figures, in which like references indicate the same or similar elements.

[0015] Figure 1 An interface of an online conference application is shown, illustrating a first example of audio / visual information that may be presented in the interface.

[0016] Figure 2 Another interface of an online conference application is shown, illustrating a second example of audio / visual information that may be presented in the interface.

[0017] Figure 3 An example of a network environment for implementing real-time audio and video conferencing or presentations is shown.

[0018] Figure 4 A general approach according to the technology for content recovery in real-time online meetings is shown.

[0019] Figure 5 is a ladder diagram illustrating the progression of data between a source or presenter participant's device and conference participant devices in an embodiment.

[0020] Figure 6 It is shown in Figure 4 A flowchart of one embodiment of a method for selecting a recovery scheme and replaying a rendering.

[0021] Figure 7 is a ladder diagram illustrating the progression of data between a source or presenter participant's device and conference participant devices in an embodiment.

[0022] Figure 8 Shows the use Figure 7 The message delivery implementation method is Figure 4 An implementation of a method for selecting a recovery scheme and replaying rendering.

[0023] Figure 9 Shown is shown in Figure 3 The network environment in which the potential cache location for recovering data is determined.

[0024] FIG10 shows the conventional transmission order of ten segments / packets from a single queue, where no recovery data is provided.

[0025] Figure 11 The transmission sequence of ten segments / packets of real-time recovered data from a single queue is shown.

[0026] Figure 12 The transmission order for real-time recovery data when multiple sub-queues are used is shown.

[0027] Figure 13 is Figure 3 or Figure 9The flowchart of the method for determining whether to clear the restoration data cache is shown and is executed at any network entity having the restoration data cache.

[0028] Figure 14 It can be done by Figure 3 and Figure 9 Flowchart of a cache scheduling algorithm for management performed by a management server or network node.

[0029] FIG. 15 shows the current status of source transmission and client reception of real-time data for a real-time online conference.

[0030] Figure 16 is a timing diagram showing the timing of segments / messages transmitted between a source participant device and a receiving participant device.

[0031] Figure 17 It shows that when Figure 16 Flowchart of the process operated on the receiving device when receiving data in the manner shown.

[0032] Figure 18 is a timing diagram showing the timing of segments / messages transmitted between source participant devices and client queuing and playback, where the recovery data message is based on Figure 7 and Figure 8 The transmission is carried out in accordance with the implementation method.

[0033] Figure 19 It shows that when Figure 18 Flowchart of the process operated on the receiving device when receiving data in the manner shown.

[0034] Figure 20 The relative timing of one example of how client device data queuing and playback corrects for lost data in embodiments herein is shown.

[0035] Figure 21 and Figure 22 Two alternative implementations of recovery process selection and processing are shown.

[0036] Figure 23 A demonstration of recovered data converted during a real-time meeting is shown.

[0037] Figure 24 is a ladder diagram showing the progression of data between a source or presenter participant's device and conference participant devices, where multiple segments or messages are lost.

[0038] Figure 25 is a flow chart illustrating a method for caching an entire real-time conference content stream.

[0039] Figure 26 Shows that it can be Figure 3and Figure 9 The cache scheduling algorithm managed by the management server or network node.

[0040] Figure 27 The timing of segments / messages transmitted between a source participant device and a recipient device and the recipient device playback of the converted data messages is shown.

[0041] Figure 28 Shows the timing of segments / messages transmitted between source participant devices and receiver devices and the use of Figure 21 The delay introduced by the conversion recovery data method.

[0042] Figure 29 An example of a replay-optimized segment / packet of data W(n) is shown.

[0043] Figure 30 The timing of segments / messages transmitted between a source participant device and a recipient device and the recipient device playback of the playback-optimized data segments / messages is shown.

[0044] Figure 31 The relative timing of one example of how client device data queuing and playback corrects for lost data in the present technology is shown.

[0045] Figure 32 is a flow chart of a general method performed at a participant device for implementing participant device-controlled data recovery.

[0046] Figure 33 Shown Figure 32 One implementation of step 3260.

[0047] Figure 34 Shown Figure 32 Another implementation of step 3360.

[0048] Figure 35 A method for participant-side recovery using network device cache is shown.

[0049] Figure 36 A comparison of the relative timing of segments / messages transmitted between a source participant device and two receiving participant devices is shown.

[0050] Figure 37 shows the relative timing of segments / messages transmitted between a source participant device and two client devices Ca and Cb, using Figure 34 Rendering method.

[0051] Figure 38The following is a general overview flow chart illustrating various implementations of proactively initiating real-time content recovery in an online meeting in conjunction with the real-time content recovery solution described herein.

[0052] Figure 39 A flow diagram illustrating local caching and network caching that may be used with proactively implemented content recovery.

[0053] Figure 40 Shown based on Figure 39 The caching approach shown enables two types of content rendering when a proactive, user-initiated interruption occurs.

[0054] Figure 41 Illustrated is content queuing and playback for an actively interrupted client device when using converted resume data.

[0055] Figure 42 Illustrated is content queuing and playback for an actively interrupted client device when using playback optimized resumption data.

[0056] Figure 43 An interface for an online conferencing application is shown, showing an example of simultaneously rendered real-time audio / visual information and recovered data.

[0057] Figure 44 Shows when using Figure 43 The interface and implementation method are the relative timing of real-time and recovery data messages.

[0058] Figure 45 is a flow chart illustrating a method for content-aware active participant interruption.

[0059] Figure 46 is a block diagram of a network processing device that can be used to implement various embodiments.

[0060] Figure 47 is a block diagram of a network processing device that can be used to implement various embodiments of a conference server or network node.

[0061] Figure 48 is a block diagram illustrating an example of a network device or node, such as Figure 3 Those network devices or nodes shown in the network.

[0062] Figure 49 An embodiment of a network message implementation method for achieving real-time data recovery is shown. DETAILED DESCRIPTION

[0063] The present disclosure and embodiments are directed to improving the performance of real-time online audio-video conferencing and ensuring better quality of experience (QoE) by recovering data that may have been lost during a live online meeting and providing that data to any participant in the meeting whose live meeting data may have been lost. Data may be lost due to communication interference or by a participant actively interrupting their device. The described embodiments may also be applicable to broadcasting presentations to multiple participants. Any reference herein to live meetings and meeting data includes broadcast presentations, where applicable.

[0064] In one aspect, systems and methods are presented for compensating for real-time data lost in a real-time online conference. Communication interference may be caused by network interference, low network bandwidth, low network throughput, excessive network latency, network packet loss, network jitter, and / or other communication problems. Real-time content recovery includes systems and methods for resolving packet loss in the transmission and reception of media delivery, such as real-time online audio / video conferencing. Real-time content is content that is generated in real time at the source and delivered and used at the target device almost simultaneously or in near real time (subject to transmission delays and latency). Real-time content includes information (audio, video, text) created by participants in the conference, and the information itself is transmitted between devices in the conference in the form of data segments or data packets. In real-time content recovery, resuming data delivery and resuming data playback work together to ensure that lost real-time content (or rendering of which is otherwise interrupted at the participant device) is rendered to the participant device in a seamless manner, thereby ensuring a good quality of experience.

[0065] The first aspect includes transmitting recovery or "catch-up" content of a real-time online audio-video conference to one or more devices of participants in the real-time conference. The recovery content is transmitted in the form of recovery data to replace one or more segments, messages or packets of real-time audio / video conference data that have been damaged or lost (passive loss) during the initial transmission or missed by the recipient at the original playback time due to various reasons (including active loss initiated by the recipient). The recovery data segments / messages replace the following segments / messages: these segments / messages were not received at the recipient in time to be rendered on time, or could not be rendered on time by the recipient due to some other reasons, thereby resulting in an interruption in the continuous playback of the real-time content. In an embodiment, the recovery content can be transmitted or retransmitted to the recipient for rendering as recovery data in the original format or in a different format (e.g., transcoded or format converted, or recovery optimized format).

[0066] Resuming playback includes resuming the rendering of content segments / packets that are received and rendered by the recipient at a time later than the original expected rendering time of the uninterrupted continuous rendering of the real-time content. Resuming playback can be defined by two thresholds τ1 and τ2, where τ1 is a timeout threshold comprising the latest reception time to avoid obvious / perceptible errors in the real-time application, and τ2 is the latest time at which the catch-up content is received according to defined user preferences and / or system definitions so that the recipient will be able to play the catch-up content within a reasonable or predefined catch-up period (time frame T cp ) to catch up content within the live online meeting to achieve acceptable QoE during the real-time online meeting.

[0067] Another aspect of real-time content recovery is content retransmission, which is different from the existing transport or application layer message retransmission. In traditional retransmission, timely retransmission is used, that is, the retransmitted message is sent after reaching the timeout threshold τ. n The previously arrived packet with sequence number n=1 has an arrival time t* of t* < τ1, and the retransmitted packet is retransmitted within its original packet. In the recovery transmission, the recovery content is transmitted between (τ1, τ2), that is, after the timeout, and the retransmitted content format can be the original, converted, or intelligently processed for optimized playback.

[0068] Another aspect is real-time conference recovery using client device controlled technology. Caching occurs in client participant devices and / or devices in the network environment, wherein these participant devices in embodiments use on-device technology to recover interrupted real-time data.

[0069] Another aspect is real-time meeting resumption compensation when a meeting participant initiates an active pause or "break" in the meeting on their own device. After initiation by the user, caching occurs on the client participant device and / or devices in the network environment, and the meeting content can be resumed using various techniques.

[0070] As discussed herein, real-time conference content may be divided into segments, which may include one or more network messages. When referring to messages or segments, it should be understood that the principles discussed herein also apply when referring to messages or segments. Standard network protocols control the order of network messages, and when it comes to transmission order, reception order, or rendering (playback) order, timestamps and / or sequence numbers may be used to control the order of segments. As used herein, the order is described relative to the sequence number, but it should be understood that timestamps or any other segment / message sequence tracking may also be used in the described technology. All segments / messages transmitted in an ordered sequence between a source device and a receiving participant device may be referred to herein as a data "stream."

[0071] Figure 1An interface 100 for an online conference application is shown, showing a first example of audio / visual information that may be presented in the interface. Interface 100 is sometimes referred to as a "meeting room" interface, in which a video, picture, or other representation of each of the participants is displayed. Interface 100 includes a presenter window 110 showing a focus participant 120 who may be speaking or presenting, and an attendee display window 130 showing other connected attendees of the real-time conference. In this example, presenter window 110 is showing the attendees, but may also include text, video, or shared screen information, such as Figure 2 In addition, the participant display window can be arranged in different positions, such as Figure 2 It may or may not be shown. In addition, the placement of the windows may be different in different embodiments. It should be understood that in embodiments, the presenter window may occupy the entire screen. The presenter window 110 may show the real-time actions, real-time video of the presenter (speaker), while other information is displayed on another portion of the display. Figure 2 In the illustrated application interface 200, the audio / visual information 135 may be a motion video with accompanying text 145 provided in a presenter window (as shown) or in a different window in the interface. It should also be understood that while eight attendees are shown in window 130, any number of users may participate in a meeting or presentation.

[0072] Figure 3 An example of a network environment for implementing a real-time conferencing application is shown. Environment 300 includes a host processing device 310 associated with a conference host. Conference hosts typically include conference organizers, who may subscribe to a service that provides real-time conferencing services using an online conferencing application. In a real-time conference, the host is not always the source of conference data, and all participant devices may contribute to conference data. In other embodiments, the host processing device may be a standalone conference host connected to the participant processing devices via a network. Additionally, while the host processing device 310 is shown as a notebook or laptop processing device, any type of processing device may implement the role of a host processing device. Three examples of participant devices are also shown, including a tablet processing device 312, a desktop processing device 314, and a mobile processing device 316. It should be understood that there may be any number of processing devices operating as participant devices for a real-time conference, where one participant device is typically associated with one participant (although multiple participants may also use a single device). Examples of processing devices are shown in FIG. Figures 43 to 45 Shown in.

[0073] Figure 3Also shown are a plurality of network nodes 320a to 320d and 330a to 330d and conference servers 340a to 340d. The conference servers 340a to 340d may be part of a cloud service 350, which, in various embodiments, may provide cloud computing services dedicated to online conferencing applications. The nodes 320a to 320d are referred to herein as "edge" nodes because such devices are typically one network hop away from the devices 310, 312, 314, 316. Each of the network nodes may include a switch, a router, a processing device, or other network coupled processing device that may or may not include data storage capabilities to enable cached conferencing and recovery data to be stored in the node for distribution to devices using the conferencing application. In other embodiments, the network nodes may be connected to a server other than the server. Figure 3 Additional levels of network nodes beyond those shown. In other embodiments, fewer network nodes are used, and in some embodiments, these network nodes comprise basic network switches without available cache memory. In yet other embodiments, the conference server is not part of a cloud service, but rather may comprise one or more conference servers operated by a single enterprise, such that the network environment is owned and contained by a single entity (e.g., a company), with the presenters and attendees all connected via that entity's private network. The lines between devices 310, 312, 314, 316, network nodes 320a to 320d, 330a to 330d, and conference servers 340a to 340d represent network connections, which may be wired or wireless and include one or more public networks and / or private networks. Examples of node devices are shown in FIG. Figure 43 and Figure 44 As shown in Figure 3 As shown, each of the participant devices 310 , 312 , 314 , 316 can provide and receive real-time conference data through one or more of the network nodes 320 a - 320 d , 330 a - 330 d , and / or the cloud service 350 .

[0074] Figure 3 Conference data streams that can be provided to client devices 312, 314, and 316 are shown. As shown, each device can send and receive real-time conference data. Each real-time conference can include at least one real-time component in which participants speak and / or present information to others in the conference, and can also include pre-prepared stored components such as a slide presentation or a shared screen or whiteboard application.

[0075] In one example, real-time conference data 375 can be sent by a moderator or source device 310 via a network interface of the moderator's processing device and directed to a client computer via, for example, a cloud service 350 comprising one or more conference servers 340a through 340d. In this example, moderator device 310 is considered the conference data "source" device. Within cloud service 350, data is distributed based on the workload of each of the conference servers 340 and can be sent directly from the conference server to the client or via one or more of network nodes 320a through 320d and 330a through 330d. In one embodiment, network nodes 320a through 320d and 330a through 330d may include processors and memory to enable the nodes to cache data from the real-time conference or presentation. In other embodiments, the network nodes may not have the ability to cache real-time conference or resume data. In yet other embodiments, conference data can be exchanged directly between participant devices without passing through a network node, or routed between participant devices via a network node without passing through a conference server. In other embodiments, peer-to-peer communication of real-time and resume content can be utilized.

[0076] Real-time meeting content recovery

[0077] Figure 4 A general method according to the technology of content recovery in a real-time online conference is shown. At 410, one or more participants using a participant processing device are admitted to the real-time content recovery service. Admission control can be provided by the conference service provider using a conference application and can be performed on a conference-by-conference or single-user basis. At 415, a prediction of expected jitter and / or packet loss in the network environment can be optionally performed. At 415, the method can use this information to determine whether to cache recovery content, and if so, whether to cache the recovery content on a network node, conference server, local machine, or other cache-supported network device. At 420, the conference real-time content begins, and the participants in the conference share conference information in various ways. The conference is usually started by a conference host (usually one of the participants) who starts the conference using conference service 350.

[0078] Once the real-time online conference begins at 420, the method determines whether any segments / messages have not been delivered to any participant by checking for message or segment timeouts. As discussed below, this may occur at a single device, a network node, or a conference server. Step 420 may also include determining whether any segments / messages are corrupted or otherwise unsuitable for rendering at the receiving device. Although the embodiments herein may discuss "lost" segments / messages, the embodiments of real-time data recovery are equally applicable to any received non-renderable segments / messages. At 425, if segments / messages are continuously received within the timeout period, the real-time conference content is continuously rendered on the participant device at 430, and caching of segments / messages may occur. Figure 3 At one or more network devices in the real-time conference environment. Whether and to what extent caching occurs typically depends on the system configuration and the real-time data recovery implementation. If a segment / packet timeout occurs at 425, a decision is made at 435 whether to initiate a real-time recovery process. The decision at 435 is based on predefined parameters, including system policies 465, which may be defined by the real-time conference service provider, user preferences 470, and network status 475. If real-time recovery is not initiated at 435, the method returns to step 430 and continues rendering the real-time conference content, with any discarded segments / packets not rendered. If real-time recovery is initiated at 435, the system may optionally estimate the resources available to provide the recovery data for use in determining the optimal data route and restoring data from caches throughout the network environment. As will be described below, various different forms of real-time recovery are provided by this technology. Some of these methods require more resources than others, and the estimate at 440 takes into account network bandwidth 480, available computing resources at 485, and cache availability 490.

[0079] At 445, a real-time content recovery scheme is selected. Furthermore, various techniques for real-time conference content recovery are described herein, and a recovery method is selected at 445. At 450, content processing may occur if the content recovery scheme so requires. Various types of content processing are discussed below, including no content processing, converted recovery data, and optimized recovery data. At 455, the recovery data is sent to the participant processing device that lost the conference content. At 460, the next segment / message is incremented, and the method returns to step 425 to determine whether additional segments / messages are missing.

[0080] Figure 5 and Figure 7 : is a ladder diagram showing content data, restoration data, and confirmation in one example of a data protocol used in an embodiment of the present technology. Figure 5 and Figure 7 Respectively relative to Figure 6 and Figure 8 Described, which shows two different methods of resuming content delivery. Figure 5 and Figure 7 Two embodiments of replacing segment / message delivery with recovery content after a loss are shown. Figure 5 and Figure 6 In one embodiment shown, for a given lost segment / packet X(n), all packets except the lost packet / segment X(n) are delivered as usual, i.e., synchronized with the rest of the group, and the lost packet / segment X(n) is transmitted as soon as bandwidth is available to recover the data packets. Figure 7 In the embodiment, the lost message / segment X(n) and the message / segment after X(n) are buffered in the cloud or edge (designated as X(m) in this article) and are received / lost with a delay time T. r (n) is transmitted. The rendering device reaches the catch-up point τ cp When , the transmission returns to normal. Figure 6 In the embodiment, the delivery is carried out sequentially, and Figure 5 In an embodiment, the delivery is not performed sequentially.

[0081] As used herein, a "catch-up" point is a point at which playback / rendering of real-time content at a particular participant device begins to resume synchronization with the rest of the participants in the real-time conference (i.e., no loss of the originally scheduled playback time occurs or playback is delayed). Thus, a catch-up point is a point in the real-time conference sequence at which it is appropriate to skip, speed up, or omit real-time conference resumption content and return to rendering the real-time conference content, and may include (by way of example and not limitation): a point after a period of audio and / or visual real-time conference content that is stagnant and / or silent (e.g., no participant is speaking or presenting, and there is little or no change in the video content); a point in time after an agreed-upon, announced, or pre-arranged "break" in the conference during which a participant may step away from their device, or during which it is understood that the public purpose of the conference is paused for a certain rest period; a point in time specified by a participant at a participant device at which the participant indicates that any unrendered resumption data may be skipped or omitted; and / or a maximum acceptable delay T cp The time point at which it ends.

[0082] Figure 5 5 is a ladder diagram illustrating the progression of data between a source or presenter participant's device 502 and a conference participant device 510, through network nodes 504 and 508, and conference server 506. (In other implementations, peer-to-peer communication of both live and recovered content may be utilized.) Figure 5 An example is shown where one of the packets or segments is lost before reaching the participant device. Figure 5In , time increases in a downward direction along the y-axis below each device. Figure 5 , at 520, four segments / messages X(1) through X(4) of real-time conferencing data originate from source 502 and are transmitted to edge 504, service host 506, edge 508, and participant device 510, which renders the segments / messages. Service host 506 can acknowledge receipt of the segments / messages to source 502, and device 510 can acknowledge receipt to service host 506. The sequence 520 of messages X(1) through X(4) represents the successful transmission and receipt of conferencing data.

[0083] At time 525, which can be any time during the real-time conference, four additional segments / messages X(n–1), X(n), X(n+1), and X(n+2) of real-time conference data originate from source 502 and travel to edge 504, service host 506, and edge 508. Messages X(n–1), X(n), X(n+1), and X(n+2) are forwarded by host service 506 to edge 508, but at 530, segment X(n) is lost due to any of a number of network issues. Consequently, device 510 identifies that a segment / message is missing from the sequence using either standard techniques for identifying dropped messages or service host-specified techniques for identifying lost segments of conference data. Additionally, a no-content acknowledgment is received at service host 506 for segment / message X(n). At 535, the device 510 initiates a service access request 555, which is transmitted via the node 508 to the service host 506, which acknowledges the service request and provides a recovery data replacement segment / message X(m) in the form of a replacement segment / message, which is then acknowledged by the device 510. In this embodiment, X(m) is a replacement segment / message that is identical to the original real-time data segment / message X(n) that it replaces. In one embodiment, the replacement segment / message X(n) is transmitted immediately when the bandwidth at the device 510 is available for transmission. Then, following the replacement segment / message X(n), additional real-time conference data segments / messages are forwarded in the form of segments / messages X(n+K+1), where "K" represents the number of segments after the last segment / message X(n) received at the device 510. (In this example, X(n+2) is the last segment / message received at the device 510, and K is equal to 1, but as Figure 7 As shown, K can be any number.)

[0084] exist Figure 5 In the embodiment, the recovery data is forwarded from the cache on the conference service host 506, but in other embodiments, the data can be forwarded from the cache on the source device 502 or the edge devices 504, 508 and the conference service server device.

[0085] Figure 6 Is shown using Figure 5 The message delivery implementation method is used to restore the real-time conference data Figure 4 Flowchart of one embodiment of steps 445, 450, 455 and 460 of . In this embodiment, for a given lost segment / message X(n), all messages except the lost message / segment X(n) are transmitted as usual, and the lost message / segment X(n) is transmitted immediately when bandwidth is available for additional data messages. At 645, an embodiment of the selected real-time conference data recovery is to transmit the recovered data in the original segment / message format out of order when the recovered data is available, and to render the recovered data in the correct transmission order with a certain delay when the recovered data is available at the receiving device. At 647, the real-time conference segment / message is buffered at one of the network devices, as described above. At 650, based on the real-time conference data recovery at the participant device ( Figure 5 The edge device 508 or conference service server 506 may initiate recovery after an ACK for a segment / message times out. After the segment / message is lost, the lost segment / message is forwarded from a data cache in one of the network elements at 655. At 655, the recovery data segments / messages (in this embodiment) are transmitted in their original format to the participant devices that did not receive the data segments / messages immediately and out of order when the recovery data segments / messages are available, for rendering in the original order immediately when the messages are available. The method then proceeds to Figure 4 Step 460.

[0086] Figure 7 is a ladder diagram illustrating the progression of data between a source or presenter participant's device 502 and a conference participant device 510 , through network nodes 504 and 508 and a cloud hosting service 506 . Figure 7 An example is shown where multiple segments / messages are lost before reaching the participant device. Figure 7 In the example, the lost message / segment X(n) and the message / segment after X(n) are buffered in the cloud or edge to receive / lose the delay T. r (n) is transmitted. Once the catch-up point τ is reached cp , the transmission returns to normal, and therefore there is a catch-up delay T introduced in this embodiment. cc .and Figure 5 Same as in Figure 7 , time increases in a downward direction along the y-axis below each device. Figure 5 Same as in Figure 7, at 520, four segments / messages X(1) to X(4) of real-time conference data complete the successful transmission and reception of conference data between the respective devices. Time 525, which may be any time during the real-time conference, during which additional segments / messages X(n–1), X(n), X(n+1), X(n+2) of real-time conference data originating from source 502 are transmitted to edge 504, service host 506, and edge 508. Messages X(n–1), X(n), X(n+1), X(n+2) are forwarded by host service 506 to edge 508, but at 530, segments X(n), X(n+1), X(n+2) are lost due to any one of a number of network issues.

[0087] At 650, a timeout τ0 occurs, and at 635, device 510 initiates a service access request, which is transmitted via node 508 to conference host server 506, which acknowledges the service request and provides replacement segments / messages X(m), X(m+1), X(m+2), which are then acknowledged by device 510. In one embodiment, replacement segments / messages X(m), X(m+1), X(m+2) (which are cached versions of X(n), X(n+1), X(n+2)) are transmitted immediately when bandwidth is available at device 510. Additional real-time conference data segments / messages in the form of segments / messages X(n+K-1), X(n+K), and X(n+K+1) are then forwarded following replacement messages X(m), X(m+1), X(m+2).

[0088] It is worth noting that in Figure 7 In the example, since multiple segments / messages X(n), X(n+1), and X(n+2) are lost, the loss delay T r (n) will be slightly less than the catch-up delay T cc (n).

[0089] Figure 8 Shows the use of Figure 7 Implementation of message delivery Figure 4 An embodiment of steps 445, 450, 455 and 460, wherein the lost packet / segment X(n) and the packet / segment after X(n) are buffered in the cloud or edge and are received / lost for a delay of T. r (n) To transmit. Figure 7 and Figure 8 In the embodiment of the present invention, the transmission of all messages (recovery data and new conference data after the lost segment / message) is carried out in sequence, and Figure 5 and Figure 6 In some embodiments, delivery is not necessarily sequential.

[0090] In this embodiment, for a given lost segment / message or sequence of segments / messages (e.g., X(n), X(n+1), X(n+2)), all segments except the lost message / segment are transmitted as usual, and the lost message / segment X(n) is transmitted immediately when bandwidth is available for additional data segments / messages. At 845, an embodiment of the selected real-time conference data recovery is to transmit the recovered data in the original segment / message format out of order when the recovered data is available, and to render the recovered data in the correct transmission order with a certain delay when the recovered data is available at the receiving device. At 847, the real-time conference segments / messages are buffered at one of the network devices, as described above. At 850, based on the data received at the participant device ( Figure 7 The method further comprises: determining a recovery data segment / message based on a segment / message not received at 510 in the example embodiment and maintaining the new data segment / message so that the new data segment / message and the recovery data message can be transmitted to the participant device in sequence. The determination can be made based on a failure to receive and acknowledge one or more messages from the participant device and / or based on a service access request made by the participant device that did not receive the segment / message. After the segment / message is lost, the lost segment / message is forwarded from a data cache in one of the network elements at 855. At 855, the recovery data segment / message in its original format is transmitted in sequence with the new data segment / message to the participant device (510) for playback in the original sequence. The method then proceeds to Figure 4 Step 460.

[0091] It should be understood that Figure 5 and Figure 6 In the example, the service host 506 executes Figure 4 Certain steps of method 400, including, for example, steps 440 to 460, are performed by participant device 510, while steps 425 and 435 are performed. In other embodiments, any of these steps may be performed by an intermediate node, a source device, any participant device, or a dedicated network element. Figure 3 and Figure 9 In a network environment, according to the embodiment, the source device can perform device resource analysis, network resource analysis, and content analysis and / or encoding functions. In the embodiments described below, any edge device can perform network resource and analysis, content caching, and content transcoding and format conversion. In the embodiments described herein, the service host device generally performs conference control and management, network resource analysis, data caching, content transcoding and format conversion, and intelligent content analysis and processing. The client or participant device can perform device resource analysis, network resource analysis, and content decoding and monitoring.

[0092] In an embodiment, a service level agreement (in Figure 5 and Figure 7 or similar figures herein) can precede the implementation of a real-time content recovery scheme. For example, before any real-time content or recovery data content is transmitted between devices, each network device can communicate with the conference host server to initiate both a connection request and confirmation and a service request and confirmation.

[0093] Figure 9 Shown in Figure 3 The possible locations of potential caches for recovering data in the network environment. Figures 5 to 8 As shown in , the recovery data may be provided from the conference server, which caches the recovery data in a recovery cache such as cache 802. It should be understood that in Figure 9 In other embodiments, each of conference servers 340a through 340d may include a recovery cache. In other embodiments, none of conference servers 340a through 340d include a recovery cache. Additionally, each of network nodes 320a through 320d and 330a through 330d may include a recovery cache, such as recovery caches 804 and 806. In other embodiments, no network nodes or only a portion of the network nodes include a recovery cache. Client participant devices may also include a recovery cache, such as cache 808. It should be understood that in embodiments in which caching is provided at a network node or conference server, each of the client participant devices may include a recovery cache, or may not include any recovery cache. In embodiments, all participant devices may include a recovery cache to enable orderly playback of real-time data until a catch-up point τ cp , thereby eliminating the catch-up delay T cc In embodiments where caching is provided at the conference server or network node, single or multiple queue caches may be utilized.

[0094] Figure 10 shows a conventional transmission sequence of ten segments / messages 902 from a single queue, wherein no recovery data is provided. In this figure and in some of the figures herein where the sequence numbers relate to accompanying text, the segments / messages are shown by circled sequence numbers. In this embodiment, segments / messages 1 to 5 are provided in the queue and are transmitted in sequence. The segments / messages are placed in a single queue based on their sequence numbers and delivery deadlines. Five segments / messages 902 (ordered 1 to 5 in Figure 10) are forwarded from the queue in sequence for transmission. When no confirmation of receipt of one of the five messages shown at 902 is received and a timeout (indication such as Figure 5 and Figure 7 When the ACK timeout is reached (shown as a dropped segment / packet), the transmission order of subsequent packets is not changed. After the ACK timeout, any additional packets (in this case, packets 6 to 10) are forwarded in sequence without recovery data relative to the dropped packet.

[0095] exist Figure 11 In, according to Figure 5 In the embodiment shown, the recovery data segments / messages are transmitted immediately when bandwidth is available. In a single queue, as shown in FIG10 , messages 1 through 5 are provided in the queue and transmitted in order before the expected ACK timeout. After the ACK timeout, a replacement segment / message for the discarded message—segment / message number 4 in this example—can be forwarded, but the transmission order is changed and, as shown at 904 a, recovery data segment / message 4 is inserted between messages 7 and 9, as shown at 907 and 908.

[0096] Figure 12 shows the use of multiple subqueues to achieve Figures 5 to 8 The process is to restore the order of data transmission. Based on the segment / message sequence number, segment / message type and delivery deadline, the segments / messages are placed in two or more subqueues. In one embodiment, the restored data segments / messages are placed in a separate subqueue from the original data segments / messages subqueue to await transmission. The transmission order and ACK timeout are similar to those in Figures 10 and Figure 11 The transmission sequence and ACK timeout are the same as shown in Figure 12 In the example, the recovery data packet 1102 (priority 4) is placed in a separate subqueue. The priority of the subqueue is defined based on application-specific or system policy rules and user preferences. Whenever the outbound link becomes idle for transmission, the recovery data packet 1102, which replaces the message sequence number 4, is forwarded. In one embodiment, when performing a transmission, the system first searches the highest priority subqueue, which may be the recovery data subqueue, and transmits that data first.

[0097] When using multiple subqueues (one for original data and one for recovery data), the network element can use a scheduling algorithm to determine when the transmission of the recovery data should occur. Example algorithms include transmission completion time, segment size, output data rate, and priority factors. For example, assuming:

[0098] F(i) = time to complete transmission of the i-th message / segment, where F(0) = 0;

[0099] F o (i) and F cc (i) represents the transmission completion time of the i-th message / segment in the original content subqueue and the restored data content subqueue respectively;

[0100] β(i) = the packet / segment size of the i-th packet / segment in bits;

[0101] β o (i) and β cc(i) is the message / segment size in bits of the i-th message / segment in the original data content subqueue and the restored data content subqueue, respectively (wherein, in some implementations, β o (i) = β cc (i));

[0102] R = the output data rate of the current network node or cloud;

[0103] α j = Priority factor of the jth subqueue, where α j ∈[1, J] and Where J is the total number of subqueues. Then, the transmission completion time is given by:

[0104]

[0105] Note that for different weighted queuing algorithms, α j The definition of β(i) may be different and may be based on application and system policies, rules and user preferences. When the system uses a constant packet / segment size, then for all packets / segments, β(i) = β, so

[0106] F(i)=i*β*α j / R

[0107] For two sub-queues, one for original content and one for restored data content, the priority factors are α o and α cc but

[0108] and

[0109] In the case of a single active subqueue, J = 1, then

[0110]

[0111] Figure 13 Shown in Figure 3 or Figure 9 A basic method for determining whether to clear the restoration data cache is performed at any of the network entities having the restoration data cache shown. Figure 13 In this approach, the entire conference content stream X(n) is cached, with respect to n∈[1,N], where N is the total number of segments / packets. In this embodiment, when recovery is initiated, the missing content segments / packets are retrieved from the cloud / edge and delivered to the recipient device, based on system specifications and user preferences. This has the advantage of very simple cache management, but may require additional storage space compared to more managed algorithms.

[0112] The method begins at 1205 and, at 1210, the segment / message number "n" is set equal to "1." The method continuously checks for the end of the conference at 1220 and, if the conference has not ended, caches the segment / message at the network element executing the method at 1225. The segment / message number is incremented at 1230, and the method continues to cache additional segments / messages at 1225. When the conference ends, a determination is made at 1240 as to whether it is time to purge the cache. The recovery data may be stored for a period of time after the conference ends based on system policy 465 or user preference 470. If it is time to purge the cache, the cache is purge at 1250, and the method ends at 1260.

[0113] Figure 14 Shows that it can be Figure 3 and Figure 9 The cache scheduling algorithm managed by the management server or network node. Figure 14 The method utilizes the original content playback speed and the above-mentioned “catch-up period” T cp . T cp A predefined threshold is included that defines the maximum acceptable delay for resuming playback of data content according to system policy and / or user preference. In one case, the recovery delay T cc Less than or equal to the catch-up period (for n∈[1,N], T cc (n)≤T cp ). Figure 14 The method can minimize the use of storage resources in the network device. The recovery delay is based in part on the rendering or playback time T of the cached segments / packets in the data cache. ch .exist Figure 14 In the method, if the playback time T ch Exceeding the predefined catch-up period T cp , the cached data is released.

[0114] At 1410, the segment / message number "n" is set equal to 1 and the cached data segment / message number "m" is set equal to 0. At 1415, a determination is made as to whether the conference has ended. If the conference has not ended, an initial determination is made at 1420 as to whether "n" is equal to 1. If so, a cache delay T is determined at 1425 based on the sum of all segments / messages currently present in the cache. ch At 1430, if T ch Not greater than T cp, then the current message is cached by setting the cache sequence number ň (the order in which the cache is counted) equal to the sequence number "n" plus the cache number "m". At 1440, the method caches message X(4) and loops back to step 1445. At 1445, if n is equal to 1, then "m" is incremented at 1450. If "n" is not equal to 1 at 1445, then the method returns to step 1415.

[0115] At 1420, if n is equal to 1, or if at 1430 T ch Greater than T cp , the oldest message X(n) in the cache is released, and steps 1460 and 1465 (equivalent to 1435 and 1440) set the cache sequence number ň to be equal to the sequence number "n" plus the cache number "m" and cache message X(n). At 1470, n is incremented and the method returns to 1445. This loop repeats, automatically clearing the oldest segment / message until the meeting ends at 1415. Here, at 1460, it is determined based on the system policy 465 and the user preference 470 whether it is time to purge the cache at 1475. If so, the cache is cleared at 1480 and the method ends at 1485. In other embodiments, a combined cloud and edge caching algorithm can be used.

[0116] Figure 15 shows the current state of source transmission and client reception of real-time data for a real-time online conference. In Figure 15, time is shown on the X-axis and increases from left to right. Figure 15 shows a plurality of segments / messages 1502 along axis 1505, which are identified by sequence number "n" (circled) in a sequence 1 to n of the source participant device originating from the real-time conference. If all segments / messages 1 to n are received at the recipient participant device, lines 1510 and 1520 will appear identical to line 1505, except that line 1510 will be delayed by any network delay introduced during transmission, while line 1520 will be delayed by any buffering delay at the network delay and the receiving device.

[0117] However, in FIG15 , the four messages with sequence numbers 4 to 7 are not received at the participant receiver device, and therefore the participant experiences a playback gap T r (4). Once the gap has passed, there is no delay in subsequent playback. However, participant users experience the playback gap and may miss important meeting information. Typically, the time required to transmit a message is a function of network latency, which is typically equal to propagation delay plus transmission delay, which are themselves functions of link distance, link speed, message size, and bandwidth.

[0118] Figure 16The timing of the following is shown: a segment / message transmitted between a source participant device and a receiving participant device without loss, a segment / message transmitted between a source participant device and a receiving participant device with loss and recovery data, and device queuing and receiving participant device rendering (or playback), wherein the recovery data segment / message is based on Figure 5 and Figure 6 (the recovery segments / messages arrive out of order). Line 1610 shows the data segments / messages X(1) to X(n) transmitted from the source device over time. Line 1620 shows the data segments / messages arriving at the first participant device, e.g., Figure 3 and arrives slightly later than the transmission due to network delays. Line 1630 shows the arrival of the segment / message at the second participant device, where segment / message X(4) is lost in transmission. Figure 5 and Figure 6 In an embodiment, recovered data content 1635 in the form of segment / message X(4) is received at a time between segments / messages X(5) and X(6). The recipient participant device rendering / playback shown as line 1640 is configured to wait until segment / message X(4) is received before initiating playback to maintain the order of the transmitted segments / messages, but introduce a playback rendering delay. At 1650, once the conference reaches a catch-up point, the conference rendering returns to normal (synchronized with other participants). In one embodiment, when an interruption or lull in information in the data stream is detected, the recipient playback at 1640 can achieve playback speed or simply jump forward to render the real-time data as received (as further described below with respect to additional embodiments).

[0119] Figure 17 is a flow chart illustrating a process operating on a receiving device when receiving data in the manner shown at 1630. At 1710, as shown Figure 5 As shown, the participant device ( Figure 5 The device 510 in the example detects a segment / message timeout and initiates a service access request. As described above, in other embodiments, the participant device does not need to send a service access request, but the service can be automatically initiated by the network device. At 1720, the receiving participant device can pause the conference rendering and cache any new messages received before receiving the resume data. Figure 16This would include messages X(5) through X(10). ) At 1730, the resume data is received in the same format as the real-time conference data, and at 1740, the participant device renders the conference playback of the segments / messages in sequence as quickly as possible. At 1750, once a catch-up point is detected, the participant device returns to rendering the real-time conference data in sync with the other participant devices in the real-time online conference. The catch-up point can be detected by a pause in the audio, a conference break defined by the host, or other means.

[0120] Figure 18 The timing of segments / messages transmitted between source participant devices and queued and played back by client devices is shown, wherein the recovered data packets are based on Figure 7 and Figure 8 The transmission is performed in accordance with the embodiment of the present invention (the recovery segment / message and the new segment / message arrive in order). Line 1810 shows the data packets X(1) to X(n) transmitted from the source device over time. Line 1820 shows the data packets arriving at the first participant device, e.g., Figure 3 The message arrives at the second participant device 312 and arrives slightly later than the transmission due to network delay. Line 1830 shows the message arriving at the second participant device, where message X(4) is lost in transmission. Figure 7 and Figure 8 In an embodiment, other network elements ensure that the recovered data content in the form of segment / message X(4) is delivered in sequence before message X(5). Thus, the receiver participant device playback at 1840 is the same as shown in line 1830, where there is a break in the sequence of transmitted segments / messages, but a playback delay is introduced. At 1850, once the conference reaches a catch-up point, the playback returns to normal. In one embodiment, when a break or lull in information in the data stream is detected, the receiver playback at 1830 increases the playback speed or simply jumps forward to render the real-time data as received (as further described below).

[0121] Figure 19 is a flow chart illustrating a process operating on a receiving device when receiving data in the manner shown at 1840. At 1910, as shown Figure 7 As shown, the participant device ( Figure 5 Device 510 in the example (e.g., device 510 in the example) detects a segment / message timeout and initiates a service access request. At 1920, the receiving participant device may pause conference rendering and await recovery data and new real-time segments / messages. At 1930, recovery data in the same format as the real-time conference data and new real-time conference data including messages in sequence following the recovery segment / message are received, and at 1940, playback of the conference occurs using the sequence / messages in the order received. At 1950, upon detecting a catch-up point, the participant device returns to rendering the real-time conference data.

[0122] Figure 20 The relative timing of an example of how client device data queuing and playback can correct for lost data in this technology is shown. Figure 20 In the case of a single packet / segment loss; however, it will be understood that Figure 20 The principle shown is similar for multiple lost segments / messages. Line 2010 shows a series of segments / messages 2020 identified by sequence number "n" (circled). As shown therein, the sequence 2020 of segments / messages can be received out of order. In this example, segment / message sequence number 5 is delivered before sequence number 3, and segment / message sequence number 10 is delivered before sequence number 9. A buffering delay is introduced on the playback device, which ensures that even received messages can be rendered in the correct order. When the timeout period T w (4) When the sequence number of the message that has not been received is indicated, recovery is initiated, and the recovery message sequence number 4 is received after sequence number 9. The recovery buffer delay T is introduced. cb , so that message sequence numbers 4 to 10 can be rendered in order until the catch-up point. Resuming playback delays the playback gap by 2025.

[0123] Media Adaptive Lost Content Recovery in Real-time Online Conferencing

[0124] Additional aspects of the technology include processing the recovered data in various forms. Figure 21 and Figure 22 Two alternative implementations of recovery process selection (step 450) and processing (step 455) are shown. Figure 21 and Figure 22 In the method, the recovered data is provided in a data format different from the data format of the originally transmitted real-time conference content. As described below, these different data formats include a converted (or format-converted) data format and a rendering-optimized (or "optimized") data format. In embodiments, other types of different recovered data formats may be used.

[0125] Usually, in Figure 21 In the method, rather than providing the recovered content data in its original format, the data is converted into another format—referred to herein as format conversion—to produce converted recovered data. Examples of format conversion include converting audio data into text data, converting video data into a series of lower-resolution images, converting audio / visual data into text-only data, and the like. The converted recovered data includes data of a smaller size than the original data (or the original format recovered data), and in some cases, such recovered data can be rendered simultaneously with real-time data from the conference.

[0126] Therefore, in Figure 21At 2145, the recovery method selected for the lost real-time data segment / packet includes recovering the content using converted recovery data. At 2147, an initial determination is made as to whether the converted recovery content is available. In embodiments, all source transmission data may be format converted at one or more network devices in a network embodiment. If converted recovery data is not available, conversion is performed at 2155. The converted recovery data, denoted as W(n), is then forwarded to the device where the data segment / packet was lost.

[0127] exist Figure 21 In the method, the converted recovery data delivery (step 2150) is the same as above Figure 5 The delivery described is similar, except that the "recovery forwarding" of data will include the converted recovery data.

[0128] Figure 23 An example of rendering converted recovery data in a user interface during a real-time meeting is shown. In this example, lost or corrupted real-time meeting data is converted to text and overlaid on the current presenter 120, who may or may not be the person generating the transcribed audio. In other embodiments, the converted recovery data can be presented in a window separate from the presenter.

[0129] Another form of recovered data processing involves intelligent processing of the recovered data to remove from the recovered data elements of the audiovisual data that are not necessary for a complete understanding of the data in real time. Figure 22 In a method, playback-optimized resume data is created by intelligently processing discarded segments / packets to remove, compress, or otherwise optimize the data, thereby accelerating the playback rendering of the data without causing information loss to participants. For example, the intelligent processing includes removing pauses, silent periods, repeated information, and non-critical content to efficiently accelerate the playback of both the discarded segments / packets for which the resume content is generated and, in one implementation, the segments / packets following the resume content, thereby providing a faster return to real-time data rendering.

[0130] Therefore, in Figure 22In the embodiment, at 2245, the recovery method selected for the discarded real-time data segment / message includes restoring the content through playback-optimized recovery data. At 2247, an initial determination is made as to whether the playback-optimized recovery content (and, in an embodiment, the intelligently processed real-time data) is available. In an embodiment, intelligent processing can occur continuously at one or more network devices in the network embodiment for all source transmission data, and the playback-optimized data segment / message X'(n) remains cached for use in recovering the real-time data lost during transmission. If the playback-optimized data is available, the playback-optimized data is forwarded to the device where the data segment / message was lost at 2250. If the playback-optimized recovery data is not available, intelligent processing is performed at 2155. Then, at 2156, a determination is made as to whether the real-time data following the missing data also needs to be processed. This can be determined based on system policy and user preferences. If so, processing of the real-time data following the missing data occurs at 2257, and the processed real-time data is forwarded along with the intelligently processed recovery data at 2258.

[0131] Figure 24 Is shown using Figure 22 The method is a ladder diagram of the progression of data between a source or presenter participant's device 502 and a conference participant device 510 , through network nodes 504 and 508 and a cloud hosting service 506 . Figure 24 An example is shown where multiple segments / messages are lost before reaching the participant device. Figure 24 , time increases in a downward direction along the y-axis below each device.

[0132] Time 525, which can be any time during a real-time conference, during which four segments / packets X(n–1), X(n), X(n+1), and X(n+2) of real-time conference data originate from source 502 and travel to edge 504, service host 506, and edge 508. Packets X(n–1), X(n), X(n+1), and X(n+2) are forwarded by host service 506 to edge 508, but at 2430, due to any of a number of network issues, segments / packets X(n), X(n+1), and X(n+2) are lost. Device 510 thus recognizes the segment / packet loss and, at 2435, initiates a service access request (or a service access request is automatically generated by another network device after an ACK reception timeout). The service access request is transmitted to node 508 and conference service server 506, which acknowledges the service request and provides playback-optimized recovery data packets X'(n), X'(n+1), X'(n+2), which contain processed recovery content that maximizes the information from the real-time conference in the most compressed form possible. The playback of optimized processed packets X'(n+k), X'(n+k+1) is continued until the catch-up point is reached. Figure 24 In the example, the initial intelligent recovery data is X'(n), X'(n+1), X'(n+2) forwarded from the cache on the conference service host 506, and X'(n+k+1), X'(n+k+2) ... starts with X(n+k), X'(n+k+1) as real-time messages at the source, which are converted by the conference server 506. Therefore, even those messages generated after the data loss can be processed as optimized recovery data until the catch-up point is reached. As described herein, the above combined Figure 3 and Figure 9 One or more network devices in the discussed network environment may include a buffer for storing real-time data segments / messages.

[0133] Figure 25 is a flow chart illustrating a method for caching an entire conference content stream of X(n) segments / messages, with respect to n∈[1,N], where N is the total number of messages / segments in the real-time conference data stream. In this embodiment, when recovery is initiated, intelligently processed segments / messages or converted content segments / messages are obtained from the cloud / edge and delivered to the recipient device based on system policy 465 and user preferences 470. Figure 25 In the example, steps 2510, 2514, and 2518 are equivalent to Figure 141210, 1220 and 1225. Once X(n) is cached, format conversion processing may occur at 2522 to generate converted recovery data in the form of converted segment / message W(n), which is cached at 2526. Additionally or in the alternative, intelligent processing for generating an optimized replacement segment / message X'(n) is performed at 2530 to generate optimized recovery data, and segment / message X'(n) is cached at 2534. The segment / message number "n" is incremented at 2541, and the method loops to step 2514 to determine whether the conference has ended. When the conference has ended at 2514, the equivalent of Figure 12 Steps 2546, 2550 and 2554 of steps 1240, 1250 and 1260.

[0134] Figure 26 Shows that it can be Figure 3 and Figure 9 The cache scheduling algorithm managed by the management server or network node. Figure 26 The method utilizes the original content playback speed and the above-mentioned “catch-up period” T cp Similarly, T cp A predefined threshold is included that defines the maximum acceptable delay for resuming playback of data content according to system policy and / or user preference. In one case, the recovery delay T cc Less than or equal to the catch-up period (for n∈[1,N], T cc (n)≤T cp ).

[0135] Steps 2610, 2615, 2620, 2625, 2630, 2635, 2640, 2645, 2650, 2655, 2660, 2665, 2670, 2675, 2680 and 2685 correspond to steps 1310, 1315, 1320, 1325, 1330, 1335, 1340, 1345, 1350, 1355, 1360, 1365, 1370, 1375, 1380 and 1385 of Figure 15, respectively.

[0136] The algorithm of Figure 15 and Figure 26The difference between the algorithms is that after the determination to cache X is made, a determination is made at 1340 or 1365 (and n is incremented at 1370) as to whether to cache the converted recovery data at 1382 or the optimized recovery data at 1392. If the converted recovery data is to be cached, then at 1384, format conversion occurs to create converted recovery data segments / packets W, which are cached at 1386. If the playback-optimized recovery data is to be cached, then at 1394, intelligent processing occurs and the playback-optimized recovery segments / messages X' are cached at 1396. The method then returns to 1345 and continues to loop through step 1315 until the meeting ends.

[0137] Figure 27 The timing of segments / messages transmitted between the source participant device and the receiver device and the receiver device playback of the converted data message W(n) is shown. Figure 27 As shown, the converted recovery data packets W(n) are transmitted and arrive out of order. Line 2710 shows the data packets X(1) to X(n) transmitted from the source device over time. Line 2720 shows the data packets arriving at the first participant device, such as Figure 3 The message arrives at the second participant device 312 and arrives slightly later than the transmission due to network delay. Line 2730 shows the message arriving at the second participant device, where messages X(4) to X(7) are lost in transmission at 2850. Figure 22 In an embodiment, the converted recovered data content in the form of segments / messages W(4) to W(7) is received at a time between segments / messages X(8) and X(n). The playback rendering of segments / messages W(4) to W(7) by the recipient participant device can begin immediately at 2780 because the converted data can be superimposed on the real-time conference data (e.g., Figure 22 in another format.

[0138] Figure 28 Shows the timing of segments / messages transmitted between source participant devices and receiver devices and the use of Figure 21 The delay introduced by the conversion recovery data method. Figure 28 The arrival of the message on line 2810, the rendering of the data on line 2820, and the confirmation of the cloud at line 2830 are shown. Figure 28 In, with Figure 27 As in the example, message sequence numbers 4 to 7 (X(4) to X(7)) are lost, and the converted recovery data message sequence numbers 4 to 7 are lost at time τ v 4. The playback start time starts with the arrival of the converted message sequence number 4 (W(4)), and the delay is reduced by the superposition of the converted data with the real-time data of the real-time messages 8 to 11. Figure 28 As shown, the buffer delay T can be restored at 2840. cb The receiver playback of the converted data occurs, and since the recovered data can be displayed together with the real-time conference data (message sequence numbers 8 to 20), the recovery catch-up delay T cc Relatively small.

[0139] Figure 29 An example of a playback-optimized segment / packet of data X'(n) is shown. Figure 29 An example of an audio waveform in an original segment is shown. As shown therein, by cropping and removing silent or near-silent segments 2902, 2904, the audio data is significantly reduced. Similar optimizations can be applied to video data and can be based on the audio data. For example, where the video data is synchronized with the audio data and there are silent periods in the audio data, if the video data does not contain meeting information, intelligent processing can remove the portion of the video data associated with the silent periods in the audio. In another example, where the video data is simply a slide presentation made by the source, the playback-optimized message can capture the slide images rather than the video including the slides, thereby significantly reducing the recovery segment / message size.

[0140] Figure 30 The timing of segments / messages transmitted between the source participant device and the receiver device and the receiver device playback of the playback-optimized data segment / message X'(n) is shown. Figure 31 As shown, X(4) is lost. Once one or more packets are lost, playback optimized data segments / packets X'(n) (in this case, playback optimized data segments / packets X'(4) to X'(8)) are received by the device that suffered the lost segments / packets. Line 3010 shows data segments / packets X(1) to X(n) transmitted from the source device over time. Line 3020 shows the segments / packets arriving at the participant device, where packet X(4) is lost in transmission at 3050. According to Figure 23 In an embodiment, playback-optimized data segments / messages X'(4) to X'(8) are received at a time after segment / message X(4), and rendering (line 3030) occurs at normal playback speed until segment / message X(9) - which is at the catch-up point, thereby enabling the recipient device to synchronize with other conference participants. The rendering of optimized messages X'(4) to X'(8) in this embodiment is shown as occurring at normal speed, but participants may have a different conference experience during the recovery period due to choppy audio or video data caused by the removal of quiet portions of the stream.

[0141] Figure 31 The relative timing of an example of how client device data queuing and playback can correct for lost data in this technology is shown. Figure 31 The timing of segments / messages transmitted between source participant devices and receiver devices and the use of Figure 22 The delay introduced by the conversion recovery data method. Figure 31 The arrival of the message on line 3110, the rendering of the data on line 3120, and the confirmation of the cloud at line 3130 are shown. Figure 31 In the example, message sequence numbers 4 to 7 (X(4) to X(7)) are lost, and the converted recovery data message sequence numbers 4' to j are lost at time τ V (4). The playback start time starts with the arrival of the playback optimized sequence number 4' (X'(4)), and the delay is slightly longer than the converted data because the optimized data must catch up with the real-time data of the real-time message. The playback start time starts with the arrival of the playback optimized data message X'(t), and the catch-up delay is reduced because the converted data is superimposed on the real-time data of messages 8 to 11.

[0142] Client-side adaptation for real-time conference data recovery

[0143] Another aspect of the present technology includes real-time conference resumption using client device-controlled techniques. In the real-time data resumption implementations described above, the rendering of the resumed data typically occurs at the same rate or speed as the real-time data segments / messages. In implementations, additional control over the resume rendering on the participant devices may be utilized. In general, in addition to the resume rendering playback scheme including playback at normal speed, accelerated playback with real-time resumption and / or forward jump real-time resumption at the recipient device may also be utilized.

[0144] Figure 32A general method for implementing participant device-controlled data recovery performed at a participant (receiving) device is shown. In the following embodiments, X*(n) represents a processed segment / message (e.g., converted segment / message W(n) or playback-optimized segment / message X'(n)) according to any of the embodiments described herein. At 3210, the real-time content of the conference begins, and the participants in the conference share the conference information as discussed herein. Once the real-time online conference begins at 3210, the method determines at 3215 whether any segments / messages have not been delivered to the device by checking the segment / message sequence number and the segment / message timeout. At 3215, if segments / messages are continuously received within the timeout period, the real-time conference content is continuously rendered on the participant device at 3225. If a segment / message is lost at 3215, a decision is made at 3220 as to whether to initiate real-time recovery on the participant device. If not, the participant device continues rendering the real-time conference content at 3225 without the discarded segments / messages. If real-time content resumption is initiated at 3220, then at 3240, a participant device controlled resume playback scheme is selected. An example of a playback method is described in Figure 33 and Figure 34 The decision made at 3240 is based on predefined parameters including system policies 465, user preferences 470, and resource availability 475, which may be defined by the conference service provider. At 3260, participant device client recovery is performed until a catch-up point is reached, at which point normal rendering of the real-time conference stream occurs at 3270.

[0145] In the first method, the participating receiver devices can play back the nth segment X(n) or X*(n) and all subsequent segments X(n+1), X(n+2), etc. or X*(n+1), X*(n+2), etc. at the normal rendering rate or speed, but with a delay τ D The recipient participant device can then catch up with the real-time conference activity with the remaining conference participants at a catch-up point (e.g., during an audio pause, conference interruption, speaker change, or other detected time point). In embodiments, the recipient participant device can skip ahead, bypassing certain content segments / messages that the participant is not interested in. Skipping can be performed automatically based on user preferences or manually by the participant. The catch-up point can be during or after the skip-forward operation.

[0146] In another participant device recovery method, the recipient participant device can play back the nth segment X(n) or X*(n) and multiple segments "K" after X(n) or X*(n) - X(n+1), X(n+2),..., X(n+K) or X*(n+1), X*(n+2),..., X*(n+K) at an accelerated rendering rate that is greater than (faster than) the normal rendering rate. In this embodiment, the participant device can synchronize data with the remaining conference participants after K segments. In this embodiment, forward jumps can also be performed. Assuming that the playback speed is set to SRp times the original speed Sp based on system specifications and / or user preferences, where SRp>Sp, then for a single segment / message loss, For example, if SRp=1.25xSp, then K=4, and for multiple segments / messages lost, (where H is the number of lost segments / packets).

[0147] This participant-side recovery has the advantage of requiring only minimal additional computing resources and can be used with any of the recovery data formats and delivery schemes discussed herein (raw, transformed, and playback-optimized).

[0148] Figure 33 Shown Figure 32 One embodiment of step 3260 of . At 3320, based on user preferences and system settings, an initial determination is made as to whether the method will use playback skips (jumps). If so, at 3330, the message sequence number "n" and the number of segments after X(n) "k" are incremented, and the number of lost segments / messages "H" is decremented. Once the number of segments after X(n) and the number of lost messages are the same, normal streaming is resumed at 3380. If no skip is made at 3320, then at 3350, playback occurs at the original speed until a catch-up point is reached at 3360. Until the catch-up point is reached at 3360, the message sequence number "n" and the number of segments / messages after X(n) "k" are incremented as playback rendering occurs.

[0149] Figure 34 Shown Figure 32Another embodiment of step 3360 of . At 3420, based on user preferences and system settings, an initial determination is made as to whether the method will use playback skipping (jumping). If so, at 3430, the segment / message sequence number "n" and the number of segments / messages after X(n) "k" are incremented, and the number of lost segments / messages "H" is decremented. Once the number of segments / messages after X(n) and the number of lost segments / messages are the same, normal streaming is resumed at 3480. If no skipping is performed at 3320, then at 3450, playback occurs at an accelerated (second / faster) speed SRp until reaching at 3460 Until 3460 As playback rendering occurs, the segment / packet sequence number "n" and the number "k" of segments / packets following X(n) are incremented.

[0150] In embodiments, participant-side recovery may be used with both local caching and / or caching at one or more of the network devices in the network environment. Figure 35 A method for participant device recovery using a network device cache is shown. At 3510, a determination is made as to whether the complete catch-up content exists in the local cache. If so, then playback of the catch-up content can begin. If not, at 3520, a service request is sent to the network device to resume the content. Beginning at 3540 with sequence number n=1, the method first determines whether X(n) is in the local cache, and if so, sends X(n) to the playback queue at 3570. If not, X(n) is obtained from the network device at 3560 and sent to the playback queue at 3570. If the resumed content is not complete, then the sequence number is incremented at 3590, and the method continues until the resumed content is fully played back at 3580.

[0151] Figure 36 36 shows a comparison of the relative timing of segments / packets transmitted between a source participant device and two receiving participant devices Ca and Cb. Line 3610 shows data packets X(1) to X(n) transmitted from the source device over time. Line 3620 shows the data packets arriving at the first participant device Cb, e.g., if no packets are lost. Figure 3 312 in the participant device and arrives slightly later than the transmission due to network delays. Lines 3630 and 3640 compare the segments / packets arriving at the second participant device Ca with the packet X(4) lost in transmission. At line 3630, the recovery data segment / packet X(4) arrives after the real-time data packet X(5), while at line 3640, the segments / packets are shown to arrive in order. At line 3650, the rendering order for participant device Cb is shown and follows the order of segment / packet reception at 3620 (after network and buffering delays).

[0152] Lines 3660 and 3670 compare the cases with and without a forward jump. Figure 36 3660, participant device Ca utilizes a forward jump to skip segments / messages X(6) and X(7) to quickly catch up in speed to segment / message X(8) at catch-up point 3680. It will be noted that participant Ca's rendering is now synchronized with participant device Cb's rendering in line 3650. Line 3670 shows playback without the forward jump, and participant device Ca will not resynchronize rendering with participant device Cb until a pause or interruption occurs at 3690.

[0153] Figure 37 shows the relative timing of segments / messages transmitted between a source participant device and two client devices Ca and Cb, using Figure 34 Lines 3910, 3920, and 3930 are equivalent to Figure 39 37. The segment / message transmission and delivery in [ 37. Line 3740 shows participant device Cb not using accelerated playback or forward jumps. Line 3750 shows device Ca using an accelerated playback rate of 1.25 times the normal rate. For one lost message, five playback-optimized messages are used until device Ca is in the same synchronization as device Cb. Accelerated playback occurs until catch-up point 3720, at which point playback is synchronized with the other participant devices.

[0154] It should be understood that Figure 36 and Figure 37 Thus, in one embodiment, both skipping and accelerated playback can be used to reach a catch-up point.

[0155] Real-time conference data recovery after active participant interruption

[0156] The real-time content recovery techniques discussed in this article have so far focused on content recovery caused by data loss or corruption due to network issues. These recovery techniques can also be applied based on active actions taken by participants at the receiving device, allowing participants to proactively pause or take a break in a real-time online meeting and later resume the missed content in a multi-person real-time online meeting.

[0157] In one aspect, the active content recovery method is Figure 4 The method is the same as that of FIG. 1 , except that steps 425 (detecting segment / message loss by timeout) and 435 (initiating real-time recovery) are initiated by the participant.

[0158] Figure 38The following is a general overview flow chart illustrating various implementations for proactively initiating real-time content recovery in an online meeting in conjunction with the aforementioned recovery schemes, including recovery using original format recovery data, playback-optimized recovery data, converted recovery data, and participant device-compensated real-time content recovery implementations. Once a participant initiates an interruption by generating a rendering interruption, the real-time data can be cached on the device itself as recovery data (in any of the formats disclosed herein) or retrieved from caches on one or more network devices to initiate recovery of the real-time meeting content. In implementations, a single subqueue on the client device can be used to queue data, or multiple subqueues can be used. Queuing can also be divided between local caches and caches on network devices. Messages / segments can be placed in two or more subqueues based on sequence number, segment / message type, and delivery deadline. In one implementation, recovery messages / segments are placed in a subqueue separate from the original data message / segment subqueue. As in other implementations, the priority of the subqueues is defined based on application-specific or system policies and rules, as well as user preferences. Intelligent queuing and caching can also be used, where queuing and caching are based on system policies and user preferences.

[0159] Reference Figure 38 At 3802, a user initiates an active interruption of a real-time online conference on their participant device. An active interruption or pause by a user may result in a rendering interruption on the user's participant processing device. The interruption ends when the user initiates a restart, where the time period between the interruption and the restart includes the pause. At 3804, data rendering on the client device is paused. At 3806, the last playback segment / message sequence number "L" is recorded. At 3808, a determination is made as to whether resume processing should not be enabled. If not enabled, the current session is terminated at 3810, and the user will need to restart or rejoin the meeting.

[0160] If recovery is initiated at 3808, available resources are estimated at 3812. At 3814, the total number of available subqueues J is set equal to the total recovery message sequence number j, and the buffering delay T is set equal to cb Set to 0 and the delay gain T caused by removing all empty segments / packets cgis also set to 0. At 3816, the number of subqueues is incremented by 1. At 3818, the type of recovery scheme to be used is selected from among the various embodiments described herein. If raw data format recovery is selected, then at 3828, the method will buffer the next data packet X(j), and if the interruption has not yet ended at 3830, the buffering delay will be increased at 3832, and the number of buffers will be incremented by 1 at 3834. If converted data recovery is utilized, then at 3820, the method will request the converted data packet W(j) and cache it, and if the interruption has not yet ended at 3822, the buffering delay will be increased at 3824, and the number of buffers will be incremented by 1 at 3826. If intelligent data recovery with optimized playback is selected, then at 3836, if segment / packet X(j) is not empty, the system will buffer X(j) and continue to check whether the interruption has ended at 3842. If the interrupt has not ended at 3842, then the buffer delay T is set at 3844. cb The buffer delay t(j) will be incremented. If X(j) is empty at 3836, then the buffer delay T at 3840 cb The buffer delay t(j) is incremented. When the interrupt ends at any of steps 3842, 3830, and 3822, data rendering begins at 3848, processing packets X(j) in sequence, and incrementing the number of buffered packets j at 3850 until all packets are removed from the buffer.

[0161] Figure 39 is a general overview flow chart showing local caches and network caches that can be used with actively implemented content recovery. In the first caching scheme, where the local cache includes all local caches, the receiving participant device, upon receiving the "interruption" initiation instruction, should cache subsequent content streams X(n), X(n+1), ..., until the "interruption end" instruction is received or until a predefined period is reached. In the second scheme, edge cache or joint edge / device cache is used. When the recipient device receives the "interruption" initiation instruction, it should request the edge server / network node to cache all subsequent content streams X(n), X(n+1), ... or should work with the edge server / network node to cache all said subsequent content streams until the "interruption end" instruction is received or until a predefined period is reached. Therefore, the estimated cache size (Chbrk) required for interrupted content buffering is

[0162] At 3902, the interrupt mode is initiated by the user. If active recovery is initiated, then at 3906, active recovery begins. If active recovery is not initiated at 3904, then at 3908, the cache size Ch used for content buffering is estimated at 3908. brkAt 3910 the sequence number is set to 1 and at 3912 a calculation is made regarding the estimated local available cache size Ch dev Is it greater than or equal to the estimated required buffer size Ch brk If yes, then at 3914 it is determined whether the local cache will be used. If yes, then at 3918, the system caches the real-time data packet at 3918 as long as the interrupt at 3916 remains valid. The sequence number is incremented at 3920, and when the interrupt at 3916 ends, playback or rendering is resumed at 3922. If at 3912 the local available cache size is not greater than or equal to the estimated required buffer size, or if it is determined at 3914 that the local cache is not to be used, then at 3924 the joint edge device cache is used. As long as the interrupt at 3926 has not ended, a check is made at 3928 to determine if the local available cache size is full. If Ch dev If the local buffer is not full, the real-time media data packet is cached locally at 3932, the sequence number is incremented at 3934, and the system loops back to step 3926. If the local buffer is full at 3920, the real-time data is cached in one of the network devices. In one embodiment, at 3920, the real-time media data packet is cached in a network device closer to the execution Figure 39 Caching occurs at the device at the network location where the participant device of the method is operating. When the interruption ends at 3926, rendering is resumed at 3922.

[0163] Figure 40 Shown based on Figure 39 The caching method shown in the figure renders two types of content when an active, user-initiated interruption is initiated. It should be noted that in the case where all original format recovery data is cached locally in the participant device, the participant device does not need to send service requests to other devices in the network. In the embodiment using both local cache and network cache, similar to the Figure 5 The service request sent by the device 510 in the example is sent to the network device that processes the resume data for the participant device. In other embodiments, even when using local cache, the receiving participant device can notify other devices of the initiated active interruption to let other participants know that the receiving participant device is temporarily suspended.

[0164] Line 4010 shows that a segment / message (sequence number only) is received at a participant device. Interrupt 4012 is initiated when segment / message sequence number 3 is received and ends at sequence number 7. Original format conference data recovery is initiated at the end of the interruption (e.g., according to the above Figure 20 methods discussed elsewhere).

[0165] The first rendering scheme shown at line 4030 assumes that all recovery data is cached locally, and therefore the local buffer is used to provide the recovery data in its original format until the segment / packet is rendered and the rendering reaches the catch-up point τ at 4027 cp Introducing buffer delay and restoring buffer delay T on the receiving device cb , which is τ v (4) Subtract τ u (4), so that sequence numbers 4 to (k–1) can be rendered in order (where “k” is the number of the catch-up point τ cp The next real-time message is then rendered in sequence).

[0166] In the second rendering scheme shown at line 4040, the local cache and the network cache cooperate to provide the recovered data in its original format until the fragment / packet is rendered and the rendering reaches the catch-up point τ at 4028 cp As shown at line 4040, any number of cached original format data segments / packets with sequence number "j" will be rendered together with the local packet. In addition, a playback buffer delay and a recovery buffer delay T are introduced on the receiving device. cb , which is τ v (4) Subtract τ u (4), so that the sequence numbers 4 to j to (k-1) are rendered in order. As shown at 4042, the network device receives the message receipt confirmation in the order of the sequence numbers of line 4010.

[0167] Figure 41 The following illustrates client device content queuing and playback for an active interruption when using converted resume data. When converted resume data is used with an active interruption, the receiving participant device can be configured to prepare the converted resume data locally. In other embodiments, the converted resume data is created at one or more network nodes. Thus, when an active interruption begins, the receiving device will issue a service request. Figure 41 Implementations are shown in which transformed recovery data and playback-optimized recovery data are created at one or more network nodes.

[0168] Line 4110 shows that the converted message (sequence number only) is received at the participant device. As shown therein, the converted data 4150 is received at sequence number 7 after the interruption 4112 ends and after the real-time conference messages 8 to 10 are received (in transmission from the source device). The converted recovery data covering the interruption period and the plurality of real-time segments / messages not rendered at the participant device is determined and received from one or more network devices (in this embodiment). When the real-time segment / message (segment / message sequence numbers 8 to n) is received at τ vWhen rendered, the converted data begins to render data on these real-time segments / messages and displays data synchronously with other participants. As shown at 4142, the network device receives message receipt confirmation in the order of the sequence numbers of line 4110, followed by confirmation for the recovered data.

[0169] Figure 42 The following illustrates client device content queuing and playback for an active interruption when using playback-optimized resumption data. When playback-optimized resumption data is used with an active interruption, the receiving participant device can be configured to prepare the playback-optimized resumption data locally. In other embodiments, the playback-optimized resumption data is created at one or more network nodes. Thus, when an active interruption begins while playback-optimized data is cached at other devices, the receiving device will issue a service request. Figure 42 Implementations are shown in which transformed recovery data and playback-optimized recovery data are created at one or more network nodes.

[0170] Line 4210 shows that playback optimized segments / messages (sequence number only) are received at the participant device. As shown therein, playback optimized data 4250 is received at sequence number 7 after the interruption 4212 ends and after receiving real-time conference messages 8 to 10 (already in transmission from the source device). At 4250, playback optimized recovery data covering the interruption period and multiple real-time segments / messages not rendered at the participant device is determined and received from one or more network devices (in this embodiment). The playback optimized data is received at τ v (4) starts rendering and continues until the catch-up point τ is reached cp , where the segment / packet number "j" indicates that the playback optimized data packet can be any number of packets received from the network cache. As shown at 4242, the network device receives the packet receipt confirmation in the order of the sequence numbers of line 4110, followed by the confirmation for the playback optimized recovery data.

[0171] In other embodiments, machine learning can be used to control the recovery cache of any of the above types of data.For regular real-time conferences, historical data available from previous conferences can be used to train and predict cache availability and bandwidth available to regular participants using machine learning algorithms.

[0172] In other implementations, the relative amounts of data cached at each of the cloud, edge, and client devices may be distributed differently according to one or more various cache distribution algorithms and / or user preferences.

[0173] In other implementations, when an active user pause is initiated at the client device, formatted catch-up data may be used concurrently with real-time data rendering.

[0174] Figure 43 Shown is an interface 4300 of an online conference application, which shows an example of real-time audio / visual information and recovery data rendered simultaneously. Interface 4300 includes a real-time data presentation window 4302, in which real-time content (video, picture or other representation) from a presenter or key participant is displayed. Interface 4300 includes a window 4330 showing other connected participants of the real-time conference. In this example, presenter window 4302 is showing the participants, but it can also include text, video or shared screen information. In addition, recovery data display window 4304 allows recovery data to be presented simultaneously. In an embodiment, while presenting window 4304, the original content segment (missing original X(n) content) can be displayed in silent mode, wherein the audio content is transcribed into text or summarized for display. While presenting the window, the converted recovery data (according to any format discussed herein) or playback-optimized recovery data, or any type of recovery data described herein can be displayed. In addition, the placement of the window can be different in different embodiments.

[0175] Figure 44 Shows when an active interrupt is initiated and used Figure 43 The playback interface is the timing of the segments / messages transmitted between the source participant device and the receiver device. Figure 44 The playback of t converted data segments / messages W(n) is shown, but any type of recovered data can be used in this playback illustration. Line 4410 shows the conference data message transmitted from the source, line 4420 shows the segments / messages arriving at the receiver, and line 4440 shows the receiver rendering (or playing back) the real-time and recovered data segments / messages. Figure 44 As shown, at 4430, the user initiates an interruption and one or more of the client device, network node or conference server can begin to buffer the real-time data segments / messages and convert the real-time data into recovery data. In this example, the converted recovery data packets W(n) are converted and shown in sequence. Figure 43 In an embodiment, the converted recovered data content in the form of segments / messages W(4) through W(7) is rendered simultaneously with segments / messages X(8) and X(n) at time 4480. Playback on the recipient participant device can begin immediately at 4480 because the converted (or cached real-time) data can be rendered simultaneously with the real-time data.

[0176] Figure 45Another embodiment of user-initiated interruption of a real-time meeting or presentation using content-aware interruption or pause is shown. In this embodiment, a network node or server (or the client itself) allows a meeting participant to initiate an active interruption when the participant specifies a certain type of content that the participant is less interested in. This enables the participant to catch up with the real-time meeting or presentation more quickly in some cases. At 4510, the meeting participant initiates an active interruption notification. The interruption notification can be specific to the content being delivered. sp The time period Np during which the participant is presenting content of a specific type or type. An example of a specific type of content is when the presenter is discussing a type of content that the participant is not interested in or is presenting a publicly available video clip. In another example, during a meeting in which multiple speakers deliver multiple presentations in sequence, the meeting participants may be particularly interested in some presentations but not in others.

[0177] At 4515, a conference participant may identify a type of content or specific content from which the participant wishes to take a break during their presentation. At 4520, in one embodiment, the participant's client device may send a notification to a conference server or a network node that implements content-aware interruption. At 4525, the node or server will determine whether specific content or content type is detected and, when detected, will notify the client at 4530. In one embodiment, the server or node detects the start of Np, or predicts that Np will begin soon at a specific time. Thus, the notification at 4520 may be to let the client know that it can take a break immediately or at a specific future time. At 4545, the server may cache the real-time content as recovery content in its original format, or prepare a converted, optimized, or other form of any type of recovery content of the types described herein and cache such content for use in recovery after an active interruption.

[0178] After notifying the client at 4530, the participant may choose to start the blackout immediately at 4535, or to start the blackout at a predicted time at 4540. At 4550, when the participant ends the blackout or when the rendering period for the identified content expires, the content is requested to be resumed at 4555, and the content is forwarded from the node or server to the client for rendering at 4560. In another embodiment, the server may choose to intelligently summarize the content and forward this resume data to the client.

[0179] In another embodiment, Figure 45 The method may be performed entirely on the participant device such that notification 4520, 4530 need not occur, and content detection at 4525 and recovery content caching and (optionally) conversion at 4545 may occur on the client device.

[0180] Figure 4646 is a block diagram of a network processing device that can be used to implement various embodiments. A particular network device may use all of the components shown or only a subset of the components, and the degree of integration may vary from device to device. In addition, the network device 4600 may include multiple instances of components, such as multiple processing units, processors, memories, transmitters, receivers, etc. The network device 4600 may include a processing unit 4601 equipped with one or more input / output devices (such as a network interface, a storage interface, etc.). The processing unit 4601 may include a central processing unit (CPU) 4610, a memory 4620, a mass storage device 4630, and an I / O interface 4660 connected to a bus 4670. The bus 4670 may be one or more of several bus architectures of any type, including a memory bus or memory controller, a peripheral bus, etc. The network interface 4650 enables the network processing device to communicate with other processing devices (such as those described herein) via a network 4680.

[0181] The CPU 4610 may include any type of electronic data processor. The memory 4620 may include any type of system memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), combinations thereof, and the like. In one embodiment, the memory 4620 may include a ROM for use at boot time and a DRAM for storing programs and data used when executing programs. In an embodiment, the memory 4620 is non-transitory. In one embodiment, the memory 4620 includes computer-readable instructions executed by one or more processors 1320 to implement embodiments of the disclosed technology, including a real-time conferencing application 4625A, which itself may include a rendering engine 4625b, a sequence number data storage 4625c, a real-time streaming data buffer and cache 4625d, a recovery data cache 4625e, and a recovery content service application 4625f. The functionality of the conferencing application 4625a, real-time stream data buffer and cache 4625d, recovery data cache 4625e, and recovery content serving application 4625f are described herein in various flow charts and diagrams.

[0182] The mass storage device 4630 may include any type of storage device configured to store data, programs, and other information and make the data, programs, and other information accessible via the bus 4670. For example, the mass storage device 4630 may include one or more of a solid-state drive, a hard disk drive, a magnetic disk drive, an optical disk drive, etc.

[0183] Figure 47 is a block diagram of a network processing device that can be used to implement various embodiments of conference server 340 or network node 320 or 330. A particular network device may use all or only a subset of the components shown, and the degree of integration may vary from device to device. Figure 46 In the example, similar numbers refer to Figure 46 In one embodiment, the memory 4620 includes a real-time conference service application 4615A, which includes sequence number data 4615c. The memory may also include a recovery content service application 4615b, which includes a format conversion engine 4615e and an intelligent recovery data generator 4615f. The recovery content service application 4615b responds to service requests generated by participant devices to provide recovery data under specific embodiments discussed herein. The format conversion engine 4615e generates the converted recovery data as described herein, and the intelligent recovery data generator generates the playback-optimized recovery data as described herein.

[0184] Figure 48 is a block diagram showing an example of details of a network device or node, such as Figure 3 According to one embodiment, node 4800 may include a router, a switch, a server, or other network device. Node 4800 may correspond to one of nodes 320a to 320d, 330a to 330d. A router or other network node 4800 may be configured to implement or support embodiments of the technology disclosed herein. Node 4800 may include a plurality of receiving input / output (I / O) ports 4810, a receiver 4812 for receiving messages, a plurality of sending I / O ports 4830, and a transmitter 4832 for forwarding messages. Although in Figure 48 48 is shown as being divided into an input section and an output section, but typically these will be I / O ports 4810 and 4830 for both downstream and upstream transmissions, and receiver 4812 and transmitter 4832 will be transceivers. I / O port 4810, receiver 4812, I / O port 4830, and transmitter 4832 may be collectively referred to as a network interface, which is configured to receive and send messages over a network.

[0185] Node 4800 may also include a processor 4820, which may be formed of one or more processing circuits and a memory or storage portion 4822. Storage 4822 may be implemented in various ways based on available memory technologies, and in this embodiment is shown as having a resume data cache 4870, which may be formed of volatile RAM memory such as SRAM or DRAM, and long-term storage 4826, which may be formed of non-volatile memory such as flash NAND memory or other memory technologies.

[0186] The storage device 4822 can be used to store both data and instructions for implementing the real-time data recovery techniques described herein, particularly instructions for causing the processor 4820 to perform functions such as caching the recovered data in its original data format and / or converting the recovered data to a different data format as discussed herein and caching the recovered data in the different data formats.

[0187] Other elements on node 4800 may include a programmable content forwarding plane 4828. Depending on the implementation, programmable content forwarding plane 4828 may be part of a more general processing element of processor 4820 or a dedicated portion of processing circuitry.

[0188] More specifically, the processor 4820 including the programmable content forwarding plane 4828 can be configured to implement embodiments of the disclosed technology described below. According to certain embodiments, the storage device 4822 stores computer-readable instructions executed by the processor 4820 to implement embodiments of the disclosed technology. The embodiments of the disclosed technology described below can also be implemented at least in part using hardware logic components, such as but not limited to field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOC) systems, complex programmable logic devices (CPLDs), special-purpose computers, etc.

[0189] Figure 49 An embodiment of a network message implementation method for achieving real-time data recovery is shown. Figure 49 The network message can be used to transmit the real-time conference data recovery technology to be used to the participant devices and network devices disclosed in this article. Figure 49 In the example of , two or more bits can be used to indicate up to four different schemes of the real-time content recovery technology described herein. As is generally known, the TCP / IP protocol stack 4910 commonly used in Internet communications includes an IP header, a TCP / UDP header, an application protocol header, and a data payload. A custom application layer provided between the TCP / UDP header and the data payload may include an identifier in a reserved portion of the application layer protocol header. Bit 0 can be used to indicate that real-time data recovery is to be used (e.g., data "0" for no recovery and "1" for recovery). Bits 1 and 2 can identify the type of recovery used, for example, active, original format data, converted data, or intelligent, playback optimized. This identifier can be read by any device in the network environment to indicate the type of real-time content recovery used in the system. Each device can then act accordingly based on the configuration of the network environment.

[0190] For the purposes of this document, it is noted that the dimensions of the various features depicted in the figures are not necessarily drawn to scale.

[0191] For the purposes of this document, the specification may refer to "an embodiment," "one embodiment," "some embodiments," or "another embodiment" to describe different embodiments or the same embodiment.

[0192] For the purposes of this document, a connection can be a direct connection or an indirect connection (e.g., through one or more other parts). In some cases, when an element is referred to as being connected or coupled to another element, the element can be directly connected to the other element or indirectly connected to the other element via an intermediate element. When an element is referred to as being directly connected to another element, there are no intermediate elements between the element and the other element. Two devices are in "communication" if they are connected, directly or indirectly, so that electronic signals can be transmitted between the two devices.

[0193] Although the present disclosure has been described with reference to specific features and embodiments thereof, it is apparent that various modifications and combinations may be made to the present disclosure without departing from the scope of the present disclosure. Accordingly, the specification and drawings are to be regarded only as illustrative of the present disclosure as defined by the appended claims, and are intended to cover any and all modifications, variations, combinations or equivalents that fall within the scope of the present disclosure.

[0194] The technology described herein can be implemented using hardware, software, or a combination of hardware and software. The software used is stored in one or more processor-readable storage devices described above to program one or more processors to perform the functions described herein. The processor-readable storage device may include computer-readable media, such as volatile and non-volatile media, removable and non-removable media. As an example and not limitation, computer-readable media may include computer-readable storage media and communication media. Computer-readable storage media can be implemented using any method or technology to store information such as computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include RAM, ROM, EEPROM, flash memory or other storage technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or any other medium that can be used to store the required information and can be accessed by a computer. One or more computer-readable media do not include propagation signals, modulated signals, or transient signals.

[0195] Communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a propagating data signal, modulated data signal, or transient data signal (e.g., a carrier wave or other transport mechanism), and includes any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media include wired media such as a wired network or direct-wired connection, and wireless media such as RF and other wireless media. Any combination of the above is also included within the scope of computer-readable media.

[0196] In an alternative embodiment, some or all of the software can be replaced by a dedicated hardware logic component. For example, but not limited to, the illustrative types of usable hardware logic components include field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOC) systems, complex programmable logic devices (CPLDs), special computers, etc. In one embodiment, the software (stored on a storage device) implementing one or more embodiments is used to program one or more processors. One or more processors can communicate with one or more computer-readable media / storage devices, peripheral devices, and / or communication interfaces.

[0197] It should be understood that the subject matter of the present invention can be embodied in many different ways and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to make this subject matter thorough and complete and to fully convey this disclosure to those skilled in the art. In fact, this subject matter is intended to encompass the alternatives, modifications, and equivalents of these embodiments within the scope and spirit of this subject matter as defined by the appended claims. In addition, in the following detailed description of this subject matter, many specific details have been set forth to provide a thorough understanding of this subject matter. However, it will be clear to those skilled in the art that this subject matter can be put into practice without these specific details.

[0198] Various aspects of the present disclosure are described herein with reference to the flowchart illustrations and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It will be understood that each frame of the flowchart illustration and / or block diagram and the combination of the frames in the flowchart illustration and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that the instructions executed by the processor of the computer or other programmable instruction execution device create a mechanism for implementing the function / action specified in one or more frames of the flowchart and / or block diagram.

[0199] The description of the present disclosure is presented for the purpose of illustration and description, but is not intended to be exhaustive or limited to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the present disclosure. Various aspects of the present disclosure are selected and described in order to better explain the principles and practical applications of the present disclosure and to enable those of ordinary skill in the art to understand the present disclosure and various modifications suitable for the intended specific use.

[0200] For the purposes of this document, each process associated with the disclosed technology can be performed serially and by one or more computing devices. Each step in a process can be performed by the same or different computing device as the computing device used in other steps, and each step is not necessarily performed by a single computing device.

[0201] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A computer-implemented method for recovering lost real-time online conference content, the computer-implemented method comprising: transmitting real-time conference data including conference content in a certain data format to participant devices and receiving real-time conference data including conference content in a certain data format from participant devices during a real-time online conference; determining that real-time conference data transmitted sequentially between devices during the real-time online conference has been lost; receiving real-time recovery data in a different data format, the real-time recovery data including real-time online conference content that replaces conference content in the lost real-time conference data; as well as The real-time recovery data is rendered to replace the real-time conference content in the lost real-time conference data.

2. The computer-implemented method of claim 1 , wherein: The conference content is in an audio / visual format, and the different data format includes a non-audio / visual format.

3. The computer-implemented method of claim 1 , wherein: The conference content is in an audio / visual format, and the different data formats include a playback rendering optimized format.

4. The computer-implemented method according to claim 1 , wherein: The rendering includes rendering the real-time recovery data at the same time as rendering real-time conference data received after the lost real-time conference data is lost.

5. The computer-implemented method of claim 4, wherein: The real-time recovery data includes a text transcription of the audio / visual data, and the rendering includes rendering the text transcription of the audio / visual data in the real-time recovery data at the same time as rendering real-time conferencing data received after the lost real-time conferencing data is lost.

6. The computer-implemented method of claims 1 and 3, wherein: The playback rendering optimized content includes audio / visual content having a subset of both the audio data and the visual data that were lost in the real-time conference data.

7. A computer-implemented method according to any one of claims 1, 3 and 6, wherein: The playback rendering optimized content includes audio / visual content having a subset of audio data that is less than all of the audio data in the lost real-time conference data.

8. The computer-implemented method of claims 1, 3, 6, and 7, wherein: The playback rendering optimized content includes audio / visual content with empty portions of the audio / visual content removed.

9. The computer-implemented method of claims 1 to 8, wherein: The real-time recovery data is received from a cache on a network device in a network environment.

10. A computer-implemented method for recovering real-time conference content lost during an online presentation, the computer-implemented method comprising: transmitting real-time conference data including real-time conference content in a certain data format to participant devices and receiving real-time conference data including real-time conference content in a certain data format from participant devices during a real-time online conference; generating real-time content recovery data in a different data format, wherein the real-time recovery data replaces the conference content in the real-time conference data; as well as The real-time resumption data is forwarded to at least one participant device to replace real-time conference content determined to be lost in communication with the host device, the real-time resumption data including content that replaces the lost content.

11. The computer-implemented method of claim 10, wherein: The conference content is in an audio / visual format, and the different data format includes a non-audio / visual format.

12. The computer-implemented method of claim 11, wherein: The conference content is in an audio / visual format, and the different data formats include a playback rendering optimized format.

13. The computer-implemented method according to claims 10 and 11, wherein: The generating includes converting conference content in conference data lost in communication with the host device into a transcription of the lost conference data.

14. The computer-implemented method of claims 10, 11, and 13, wherein: The generating includes generating a text transcription of the audio / visual data that is forwarded for rendering at the same time as rendering of real-time conferencing data received after the lost real-time conferencing data is lost.

15. The computer-implemented method of claims 10 and 12, wherein: The generating includes creating playback rendering optimized content having audio / visual content that is a subset of both audio data and visual data in the lost real-time conference data.

16. The computer-implemented method of claims 10, 12, and 15, wherein: The generating includes generating playback rendering optimized content having only audio data and being a subset of audio / visual content less than all of the audio / visual content in the lost real-time conference data.

17. The computer-implemented method of claims 10, 12, 15, and 16, wherein: The playback rendering optimized content includes audio / visual content with empty portions of the audio / visual content removed.

18. A processing system in a network, the processing system comprising: a storage medium comprising computer instructions; One or more processors coupled to communicate with the storage medium, wherein the one or more processors execute the instructions to cause the system to perform the following operations: Routing real-time conference data transmitted between a source device and at least one participant processing device during a real-time online conference, the real-time conference data including conference content in a certain data format to and from a participant device during the real-time online conference; generating real-time recovery data in a different data format, wherein the real-time recovery data replaces the real-time conference content in the real-time conference data; storing the real-time recovery data; and The real-time resumption data is forwarded to at least one participant device to replace real-time conference content determined to be lost in communication with the at least one participant device, the real-time resumption data including content that replaces the lost content.

19. The processing system according to claim 18, wherein: The conference content is in an audio / visual format, and the different data format includes a non-audio / visual format.

20. The processing system of claim 18, wherein: The conference content is in an audio / visual format, and the different data formats include a playback rendering optimized format.

21. The processing system according to claims 18 and 19, wherein The one or more processors execute the instructions to generate by converting conference content in conference data lost in communication with the host device into a transcription of the lost conference data.

22. The processing system according to claims 18, 19 and 21, wherein The real-time recovery data includes a text transcription of the audio / visual data that is forwarded for rendering at the same time as rendering of the real-time conferencing data received after the lost real-time conferencing data is lost.

23. The processing system according to claims 18 and 20, wherein The playback rendering optimized content includes audio / visual content having a subset of both the audio data and the visual data that were lost in the real-time conference data.

24. The processing system according to claims 18, 20 and 23, wherein The playback rendering optimized content includes audio / visual content having a subset of audio data that is less than all of the audio data in the lost real-time conference data.

25. The processing system of claims 18, 20, 24 and 23, wherein The playback rendering optimized content includes audio / visual content with empty portions of the audio / visual content removed.

26. A user equipment comprising: a storage medium comprising computer instructions; One or more processors coupled to communicate with the storage medium, wherein the one or more processors execute the instructions to cause the system to perform the following operations: transmitting real-time conference data including real-time conference content in a certain data format to participant devices and receiving real-time conference data including real-time conference content in a certain data format from participant devices during a real-time online conference; determining that real-time conference data transmitted between devices during the real-time online conference has been lost; receiving real-time recovery data in a different data format, the real-time recovery data replacing the real-time conference content in the lost real-time conference data; and The real-time recovery data is rendered to replace the real-time conference content in the lost real-time conference data.

27. The user equipment according to claim 26, wherein: The conference content is in an audio / visual format, and the different data format includes a non-audio / visual format.

28. The user equipment according to claim 27, wherein: The conference content is in an audio / visual format, and the different data formats include a playback rendering optimized format.

29. The user equipment according to claims 26 and 27, wherein: The rendering includes forwarding the real-time recovery data to be rendered at the same time as rendering real-time conferencing data received after the lost real-time conferencing data is lost.

30. The user equipment according to claim 29, wherein: The real-time recovery data includes a text transcription of the audio / visual data.

31. The user equipment according to claims 26 and 28, wherein: The playback rendering optimized content includes audio / visual content having a subset of both the audio data and the visual data that were lost in the real-time conference data.

32. The user equipment according to any one of claims 26, 28 and 31, wherein: The playback rendering optimized content includes audio / visual content having a subset of audio data that is less than all of the audio data in the lost real-time conference data.

33. The user equipment according to claims 26, 27, 31 and 32, wherein: The playback rendering optimized content includes audio / visual content with empty portions of the audio / visual content removed.

Images