Optimizing media experience in conferences with different participants

Abstract

The invention relates to optimizing media experience in a conference with different participants. In one embodiment, a participant system in an audiovisual (AV) conference may use methods to ensure that all participants have a common platform to support an AV conference, such as a common codec, and also attempt to provide a higher quality AV conference based on available bandwidth and other standards (e.g., superior to the common codec). When bandwidth or other criteria indicate that the quality of the AV conference is reduced, the participant system may use a common codec as a fallback platform. Other embodiments are also described.

Description

[0001] This application is a divisional application of the invention patent application No. 202210606471.1, titled "Optimizing Media Experience in Meetings with Different Participants", with an application date of May 31, 2022. Background Art

[0002] The field of the present disclosure relates to real-time communication typically via a computer network or a group of networks.

[0003] Real-time communication (such as video calls) using a data processing system that can provide audio and video during communication has become very common. Currently, there are many known examples of such communication, including, for example, FaceTime from Apple Inc., WebEx from Cisco, Teams from Microsoft, Zoom, etc. These video call communications allow participants to talk in real time, similar to a regular audio call, and also allow images to be sent and received while talking, so that participants can see each other or see their computer screens or see other content while talking. These communications can be similar to being present at a meeting and can be referred to as a meeting or an audio-visual (AV) meeting.

[0004] Typically, participants in a meeting use different types of data processing systems that may have different media capabilities. For example, one participant's data processing system can be a laptop computer running a certain version of the Windows operating system, and another participant's data processing system can be a smartphone running a certain version of the Android operating system, and the data processing system of a third participant can be an iMac desktop computer running a certain version of the macOS operating system. These different systems can have different media capabilities, such as different video codecs. Due to this difference, different systems are forced to use the lowest common configuration, but this will not bring the best experience for everyone in the meeting. An alternative form of using the lowest common configuration can be a method of using a server that performs media transcoding between two different systems. However, when end-to-end security is required (for example, encryption that makes the content opaque at the server), it is impossible for the server to perform media transcoding because the server cannot decrypt the content. Summary of the Invention

[0005] This patent application claims the benefit of U.S. Provisional Patent Application No. 63 / 197,214, filed on June 4, 2021, which is incorporated herein by reference.

[0006] In one embodiment, participant systems in an audio-visual (AV) conference can use methods to ensure that all participants have a common platform to support the AV conference, such as a common codec and a common encryption method, and also attempt to provide a higher quality AV conference (e.g., better than the common codec) based on the available bandwidth and other criteria of a subset of participant systems that support a better codec. The AV conference can be used with end-to-end encryption such that no server media transcoding is required and no server decryption is required, and using a common codec and a common encryption method can allow different types of data processing systems to participate in the conference. When the bandwidth or other criteria indicate a degradation in the quality of the AV conference, the participant systems can also use the common codec as a fallback platform. In one embodiment, at least some of the participant systems should be able to use a media codec that is better than the common codec; for example, when a subset of participant systems supports a better codec, those participant systems can send and receive streams encoded with the better codec and can also send and receive streams encoded with the common codec for those participants who will use the common codec (e.g., because they do not have the better codec or they fallback to using the common codec due to, for example, reduced network bandwidth). In the case of a fallback, the participant systems should not have to incur the cost of encoding and transmitting streams with a better media codec if the other participants in the conference are not able to consume those streams. Also, in the case of a fallback, a participant system that cannot maintain encoding and transmitting multiple streams (such as a common codec stream and a stream encoded with a better codec) can transmit only the common codec stream and still participate in the conference.

[0007] One aspect of the present disclosure relates to creating a common codec stream on demand in response to a participant (e.g., a third data processing system) joining a meeting. According to this aspect, a method may be performed by a first data processing system in an environment including a first data processing system, a second data processing system, a third data processing system, and one or more servers, and the method may include the following operations: transmitting, by the first data processing system, codec capabilities of the first data processing system to the server, the codec capabilities of the first data processing system indicating a first set of codecs in the first data processing system, the first set of codecs being available for use in an audio-visual (AV) meeting; receiving, by the first data processing system, a second set of codec capabilities of the second data processing system, the second set of codec capabilities indicating a second set of codecs in the second data processing system, the second set of codecs being available for use in the AV meeting; receiving, at the first data processing system, a request from the second data processing system to receive a first stream encoded with a first codec in the AV meeting, the first codec being in the first set of codecs in the first data processing system; transmitting, by the first data processing system, the first stream to the server for delivery to the second data processing system; receiving, by the first data processing system, a request from the third data processing system to receive a second stream encoded with a second codec in the AV meeting, the second codec being in the first set of codecs and different from the first codec; creating, on demand and in response to the request from the third data processing system, the second stream in the AV meeting; and transmitting the second stream to the server for delivery to the third data processing system while the first data processing system continues to transmit the first stream to the server for delivery to the second data processing system.

[0008] In one implementation of the method, each codec in the first set of codecs and the second set of codecs is configured to compress video content in the AV meeting for transmission to other participants in the AV meeting and is configured to decompress video content received in the AV meeting, and the second codec is common to the first data processing system, the second data processing system, and the third data processing system and is available for use in the first data processing system, the second data processing system, and the third data processing system. The method may support different types of data processing systems; for example, the first data processing system and the second data processing system may execute one or more versions of a first operating system (e.g., iOS from Apple Inc.) during the AV meeting, and the third data processing system may execute a second operating system (e.g., Android from Google) during the AV meeting. In this example, the first data processing system and the second data processing system may use the FaceTime application to participate in the AV meeting, and the server may be a server in the FaceTime infrastructure, while the third data processing system uses a web browser to participate in the AV meeting.

[0009] In one embodiment of the method, the first data processing system may monitor one or more of the following: uplink bandwidth from the first data processing system, downlink bandwidth to the first data processing system, thermal data regarding the thermal state of the first data processing system, or battery state regarding a battery in the first data processing system; and the first data processing system may switch from transmitting a first stream to a second data processing system to transmitting a second stream to the second data processing system in response to one or more conditions determined from the monitoring. In one embodiment, the switch may occur by associating the second stream with the stream identifier of the first stream such that the server forwards the second stream to the second data processing system. In another embodiment, the switch may occur by: transmitting an indication to the server that the first data processing system has or will stop transmitting the first stream to be delivered to the second data processing system; receiving a request for the second stream from the second data processing system; and transmitting the second stream from the first data processing system to the server to be delivered to the second data processing system.

[0010] The method performed by the server according to this aspect of the on-demand creation of a common codec stream may include the operations of: receiving, by the server, codec capabilities of a first data processing system from the first data processing system, the codec capabilities of the first data processing system indicating a first set of codecs in the first data processing system, the first set of codecs being available for use in an audio-visual (AV) conference; transmitting, by the server, codec capabilities of a second data processing system to the first data processing system, the codec capabilities of the second data processing system indicating a second set of codecs in the second data processing system, the second set of codecs being available for use in the AV conference; transmitting, by the server, a request to the first data processing system from a second data processing system for a first stream encoded with a first codec in the AV conference, the first codec being in the first set of codecs in the first data processing system; receiving, by the server, the first stream from the first data processing system to be delivered to the second data processing system; transmitting, by the server, a request to the first data processing system from a third data processing system for a second stream encoded with a second codec in the AV conference, the second codec being in the first set of codecs and different from the first codec; and receiving, at the server, the second stream to be delivered to the third data processing system while the server continues to transmit the first stream to the second data processing system.

[0011] Another aspect of the present disclosure relates to the use of a better or best codec for at least one subset of participant systems in a conference. According to this aspect, a method may be performed by a first data processing system in an environment including a first data processing system, a second data processing system, and a server, and the method may include the following operations: transmitting, by the first data processing system, codec capabilities of the first data processing system, the codec capabilities of the first data processing system indicating a first set of codecs in the first data processing system, the first set of codecs being available for use in an audiovisual (AV) conference; receiving, by the first data processing system, second codec capabilities of the second data processing system, the second codec capabilities of the second data processing system indicating a second set of codecs in the second data processing system, the second set of codecs being available for use in the AV conference; transmitting, by the first data processing system, a request for a first stream encoded by a first codec at the second data processing system, the request being based on a criterion at the first data processing system of the highest quality codec common to the first set of codecs and the second set of codecs.

[0012] Aspects and embodiments described herein may include a non-transitory machine-readable medium that may store executable computer program instructions that, when executed, cause one or more data processing systems to perform the methods described herein when executing the computer program instructions. The instructions may be stored in a non-transitory machine-readable medium, such as in dynamic random access memory (DRAM) such as flash memory or other forms of memory that serve as volatile or non-volatile memory. Aspects and embodiments described herein may also be in the form of data processing systems constructed or programmed to perform these methods. For example, a data processing system may be constructed of hardware logic to perform these methods, or may be programmed with a computer program to perform these methods, and such a data processing system may be considered a participant system or a server system that may be used in a conference, such as the AV conference described herein.

[0013] The above summary does not include an exhaustive list of all embodiments and aspects of the present disclosure. All systems, media, and methods may be practiced in accordance with all suitable combinations of the various aspects and embodiments of the above summary and those disclosed in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The present invention is illustrated by way of example and is not limited to the figures of the various drawings, in which like reference numerals indicate like elements.

[0015] Figure 1 An example of a conference environment that may be used with embodiments described herein is shown.

[0016] Figure 2Ais a flowchart showing a method according to an embodiment that creates a new stream (e.g., a stream encoded with a common codec) for a new participant system while maintaining an existing stream of another participant system.

[0017] Figure 2B shows a set of participant systems and a server example that can execute Figure 2A the method shown.

[0018] Figure 2C is a flowchart showing a method that can be executed by Figure 2B the server system in.

[0019] Figure 3A is a flowchart showing a method according to an embodiment where a participant system falls back to a second stream using a different codec such as a common codec.

[0020] Figure 3B shows a set of participant systems and a server example that can execute Figure 3A the method shown.

[0021] Figure 4A is a flowchart showing a method according to another embodiment where a participant system falls back to a second stream using a different codec such as a common codec.

[0022] Figure 4B shows a set of participant systems and a server example that can execute Figure 4A the method shown.

[0023] Figure 5A is a flowchart showing a method according to an embodiment that allows a participant system to select better codecs when other systems support better codecs.

[0024] Figure 5B shows a set of participant systems and a server example that can execute Figure 5A the method shown.

[0025] Figure 6 shows an example of a data processing system that can be used to implement a participant system or a server system (e.g., a forwarding server). Detailed Description

[0026] Various embodiments and aspects will be described with reference to the details discussed below, and the drawings will illustrate the various embodiments. The following description and drawings are illustrative and should not be construed as restrictive. Many specific details are described to provide a thorough understanding of the various embodiments. However, in some instances, well-known or conventional details are not described to provide a concise discussion of the embodiments.

[0027] As used in this specification, the phrase "an embodiment" or "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment. The phrase "in an embodiment" appearing in various places in this specification is not necessarily all referring to the same embodiment. The processes depicted in the subsequent figures are performed by processing logic components including hardware (e.g., circuits, dedicated logic components, etc.), software, or a combination of both. Although the processes are described below in accordance with certain orders of operations, it should be understood that some of the described operations can be performed in different orders. Additionally, some operations can also be performed in parallel rather than sequentially.

[0028] Audio-visual conferencing enables the reception and transmission of audio and / or video signals (typically as streaming content) by user devices or systems (e.g., at different locations) to enable real-time communication between users. In some cases, two users can communicate with each other in a one-to-one communication using audio-visual conferencing at their respective devices. In other cases, multi-party audio-visual conferencing can be utilized by more than two users to participate in a real-time group session.

[0029] In some systems of multi-party audio-visual conferencing, one or more of the participant systems in the participant systems may experience network degradation. In the one-to-one audio-visual conferencing example, both of the two participating systems can switch from a high-quality / bitrate stream to a lower-quality / bitrate stream in order to adapt to the bandwidth degradation. However, when there are more than two participants in an audio-visual conference, switching all of the participant devices to a lower-quality / bitrate content stream because a single participant device has a bandwidth constraint may cause the audio-visual conference experience of all participants to degrade. When the participant systems are different types of systems (such as an iPhone executing FaceTime on a version of iOS and a smart phone executing a certain version of the Android operating system), the switching becomes more complex.

[0030] Figure 1 An example of a conferencing environment 10 is shown that can be used with one or more of the embodiments described herein. However, not all of the depicted components can be used in all specific implementations of the embodiments, and one or more specific implementations can include additional or different components compared to those shown. Variations in the arrangement and type of components can be made in accordance with the embodiments described herein, and additional components, different components, or fewer components can be provided in accordance with the embodiments described herein.

[0031] The conferencing environment 10 includes data processing systems 15, 17, and 19, one or more computer networks 12 (e.g., the Internet), and one or more AV conferencing servers 14. One or more networks 12 are communicatively (directly or indirectly) coupled to any two or more of the data processing systems such as data processing systems 15, 17, and 19 and one or more servers 14 to permit data exchange between the systems and the one or more servers. In one or more specific embodiments, one or more networks 12 may be an interconnected network that may include the Internet and / or devices communicatively coupled to the Internet. For purposes of explanation, the conferencing environment 10 is shown in Figure 1 as including systems 15, 17, and 19 and one or more servers 14; however, the conferencing environment 10 may include any number of electronic devices or data processing systems and any number of servers, typically millions of devices or systems spread over a large geographic area. Additionally, the conferencing environment 10 may include different types of servers that perform messaging operations (e.g., transmission of push notifications or push messages between participant systems) and registration operations (e.g., a FaceTime registration server for registering users of a FaceTime AV conferencing system) and other operations for setting up conferences between participant systems such as data processing systems 15, 17, and 19. One or more servers 14 may be a server cloud for facilitating AV conferences between data processing systems (such as systems 15, 17, and 19). In one embodiment, one or more servers 14 may include the server 108 shown and described in U.S. Patent No. 10,931,725, which describes a server used in a group FaceTime architecture or infrastructure; other servers described in that U.S. patent (e.g., other selective forwarding servers) may also be used in the embodiments described in this disclosure.

[0032] The data processing systems 15, 17, and 19 can be, for example, desktop computers, portable computing devices (such as laptop computers), tablet computers (e.g., iPad), smart phones (e.g., iPhone or Android smart phones), smart speakers (e.g., Echo or Echo Show from Amazon), peripheral devices (such as digital cameras, headphones), gaming devices or systems, wearable devices (such as head-mounted displays or glasses or smart watches, etc.), or any other suitable device or consumer electronic device including, for example, one or more wireless interfaces, such as WLAN radios, WiFi radios, cellular radios, Bluetooth radios, Zigbee radios, near-field communication (NFC) radios, and / or other wireless radios. These data processing systems can be configured to participate in an audio-visual conference, for example, in which the data processing systems 15, 17, and 19 (also referred to as participant devices or participant systems) can participate in a group session in which video and / or audio content streams are transmitted between the participant devices in the AV conference. In the context of the embodiments described herein, an AV conference will be understood to mean a communication in which at least one of audio or video is transmitted as a streaming content between participant systems; typically, in one embodiment, both audio and video are transmitted (assuming at least some of the participant systems are equipped with cameras), but in some cases, when the network bandwidth degrades to a point where only audio transmission can be supported from some or all of the participant systems, only audio may be transmitted. In one embodiment, only audio transmission can occur at any point during (or even throughout) the AV conference. In one embodiment, at least some of the participant systems may have an AV conference application (e.g., FaceTime application) installed on the participant system; the AV conference application on the sending device (e.g., data processing system 15) can facilitate the transmission of the streaming content to be received by at least one other participant with the same AV conference application (or a certain version of the application) that also has the same media capabilities as the sending device. In one embodiment, one or more participants may not include a dedicated AV conference application (e.g., they do not include the FaceTime application), and they can participate in the AV conference using a web browser (or a similar application). In such cases, such participant systems can be referred to as web participants, and they can use known protocols, such as webRTC or the fast relay protocol, to participate in the AV conference. In one embodiment, the audio and video content can be encrypted with end-to-end encryption so that intermediate servers along the path cannot decrypt the content.

[0033] Reference will now be made to Figure 2A 、 2Band 2C while simultaneously describing a method according to one embodiment. In this method, a new participant system having media capabilities different from those of the existing participant systems (e.g., participant system 107, which can be a Figure 2B web participant in) joins an ongoing AV conference (or at least an initiated AV conference) between the existing participant systems, causing the existing participant systems to create a new stream with a codec common among the participant systems, while the original two existing participant systems (e.g., Figure 2B participant systems 103 and 109 in) continue to receive and send content to each other using different (e.g., better) codecs. A codec is a system (e.g., software and hardware) that compresses the content for transmission from a device and decompresses the received content for presentation at the device (e.g., display an image and / or play audio). Figure 2A shows a method performed by one of the existing participant systems (e.g., participant system 103) in the existing participant systems, and Figure 2C shows a method performed by a forwarding server (e.g., Figure 2B forwarding server 105 in). Figure 2A The method shown can start after initiating an AV conference; for example, this initiation can occur when a user at participant system 103 selects other participants via an audiovisual conferencing application and / or can initiate an audiovisual conference from different communication modes, such as group text messaging, group voice calling, etc. For example, the participants can be selected from the contacts included in the contact address book stored on participant system 103. The user can initiate the audiovisual conference by selecting appropriate user interface elements provided by the audiovisual conferencing application or the contact address book application or the text messaging application or the voice calling application, thereby prompting the invited participants to accept or reject participation in the audiovisual conference at their respective devices (e.g., participant systems 107 and 109).

[0034] In Figure 2A operation 51 in, a first data processing system (e.g., Figure 2B participant system 103 in) transmits its codec capabilities (e.g., a first set of codec capabilities) for use in the AV conference; this transmission can be directed to a selective forwarding server such as Figure 2Bat the selective forwarding server 105 (or a setup server for facilitating an AV conference) and received thereby. In one embodiment, when other participant systems join the AV conference, the transmitted content can then be forwarded by the selective forwarding server to the other participant systems. The codec capabilities (transmitted in operation 51) can be an exhaustive list of all available codecs (both audio and video) available at the first data processing system for use in the AV conference, and can also include a list of all available encryption methods (for end-to-end encryption) and other available capabilities (such as loss recovery techniques) available for the AV conference. In operation 53, the first data processing system (e.g., Figure 2B the participant system 103 in Figure 2B ) can receive a second set of codec capabilities from a second data processing system (e.g., Figure 2B the participant system 109 in Figure 2B ), and the received second set can be an exhaustive list of all available codecs (both audio and video) available for use in the AV conference, and can also include a list of all available encryption methods (for end-to-end encryption) and other available capabilities (such as loss recovery techniques) available for the AV conference. In operation 55, the first data processing system ( Figure 2B the participant system 103 in Figure 2B ) can receive a request from the second data processing system (e.g., Figure 2B the participant system 109 in Figure 2B ) for a first content (e.g., video) stream encoded with a first codec (which is one of the codecs in the set of capabilities transmitted in operation 51 and is also one of the codecs in the second set of capabilities). In one embodiment, in response to the request received in operation 55, the first data processing system can transmit the first stream in operation 57 to be delivered to the second data processing system via a forwarding server (e.g., Figure 2B the forwarding server 105 in

[0035] ). The first stream can be encoded by the highest or highest quality codec common to both the first data processing system and the second data processing system (based on, for example, content resolution or other parameters).

[0035] In Figure 2B the example shown, the participant system 103 transmits the first stream 115 to the forwarding server 105 as part of operation 57, and the forwarding server in turn forwards the first stream as stream 117 to the participant system 109, as Figure 2BAs shown. When circumstances permit, the first stream is encoded using the highest quality codec common to both the first (participant system 103) and second (participant system 109) systems (e.g., the first codec) to allow both participant systems to experience the best possible quality video or other content. Participant system 109 may also transmit the content encoded by the first codec common to both systems 103 and 109 to participant system 103 (via forwarding server 105), which is received, decompressed / decoded, and presented by participant system 103.

[0036] When a third participant (e.g., Figure 2B participant system 107 in Figure 2B joins the AV conference, Figure 2A participant system 103 in Figure 2B receives a request for a second stream encoded using a second codec (e.g., Figure 2B request or subscription 121 shown) from the third participant in operation 59 of Figure 2B The second codec may be a common codec available at all three participants ( Figure 2A participant systems 103, 107, and 109 shown in Figure 2B and may be different from the first codec and may be a lower quality codec (in terms of one or more of picture quality or loss behavior or resolution, etc.). After the forwarding server receives the request or subscription 119 from participant system 107, request 121 is forwarded by forwarding server 105. In one embodiment, the third participant may be a web participant that does not include the first codec used by participant systems 103 and 109 to send and receive the first stream 115; in one embodiment, the forwarding server may attempt to verify at least one common codec available at all participant systems included in the AV conference. In response to the request in operation 59, in operation 61, the first data processing system generates a second content stream for the AV conference encoded by the second codec that can serve as a common codec ( Figure 2B second stream 123 shown). In one embodiment, the second stream may be captured by the same camera on the first data processing system that also captures the content (e.g., video) of the first stream. In Figure 2A and Figure 2BThe method shown allows the participant systems 103 and 109 to continue using a codec better than the common codec while allowing the participant system 107 to join the AV conference. Thus, when sending and receiving content (e.g., video) from the participant system 107 using the second codec, the participant systems 103 and 109 can use the first codec to send and receive content (e.g., video) between themselves. Although Figure 2B a transmission from the participant system 103 is shown, it should be understood that the participant system 103 also receives content from the participant systems 107 and 109, and the participant systems 107 and 109 also transmit content to other systems, such that Figure 2B each of the three participant systems in

[0037] Figure 2C sends content to and receives content from the other participant systems. Figure 2A The method shown can be performed when Figure 2C the method in Figure 2C is performed by the first participant system (e.g., the participant system 103). In Figure 2B operation 151 shown, the forwarding server can receive the codec capabilities of the first data processing system (e.g., Figure 2C the participant system 103 in Figure 2B ); these codec capabilities can then be forwarded to the other participant systems in the AV conference. In Figure 2C operation 153, the server can receive a second set of codec capabilities (e.g., an exhaustive list of available encoders for the AV conference) from the second data processing system (e.g., Figure 2BThe participant system 103 in FIG. 10 transmits a request from the second data processing system to receive a first stream encoded by a first codec on the first data processing system (where the first codec is in a list of available codecs at the first data processing system). Then in operation 157, the forwarding server receives the requested first stream to deliver to the second data processing system, and the forwarding server then forwards the first stream to the second data processing system. Of course, the forwarding server may also receive a request from the first data processing system to receive a stream encoded at the second data processing system with the first codec from the second data processing system (where the codec used at the first data processing system and the second data processing system may be the same for the streams they send to each other), and the forwarding server may receive the stream from the second data processing system and forward the stream to the first data processing system.

[0038] In operation 159, the forwarding server receives a message from a third data processing system (eg, Figure 2B The use of the participant system 107) in Figure 2B , and the forwarding server transmits the request to the first data processing system and the second data processing system (participant systems 130 and 109). Then in operation 161, the forwarding server receives the requested second stream (encoded with the second codec) and forwards the second stream to the third data processing system, while the forwarding server continues to forward the stream encoded with the first codec to the first data processing system and the second data processing system. This permits participant systems 103 and 109 to continue to use a better codec (the first codec) than the common codec (the second codec), while permitting participant system 107 to join the AV conference. In one embodiment, the first data processing system and the second data processing system can be devices that execute versions of the same operating system (e.g., versions of iOS) and share the same media capabilities and use the same AV conferencing application (e.g., FaceTime), while the third data processing system can be a device that executes a different operating system (e.g., an Android operating system) and does not include the FaceTime AV conferencing application.

[0039] In one embodiment, participant systems using better codecs may continue to do so while conditions (e.g., network bandwidth, battery power, thermal conditions) allow such use; however, conditions may change and require a fallback method of switching to using a common codec. Figure 3A and Figure 3B A specific implementation of the fallback method is shown, and Figure 4A and Figure 4B Another specific implementation of the fallback method is shown.

[0040] existFigure 3A In the method shown, the participant system is not required to signal changes to the stream; rather, each participant is configured to be ready to switch to using a common codec in the fallback case when receiving content encoded with the common codec. In Figure 2B the context of the environment shown, at least one participant will already be using the common codec. In Figure 3A operation 201 in, each participant system (e.g., Figure 3B participant systems 210 and 216 in) monitors its local state, which may include upload bandwidth, download bandwidth, thermal data regarding the thermal state of the system (e.g., does the device overheat when it has sent multiple streams during an AV conference?), and the battery state of one or more batteries (e.g., has the battery been depleted below one or more thresholds?).

[0041] This monitoring may occur periodically during the AV conference. U.S. Provisional Patent Application No. 63 / 041,549, filed on June 19, 2020 (by Hsien-Po Shiang et al. and titled "HIGH FREQUENCY PROBING FOR NETWORK BANDWIDTH ESTIMATION USING VIDEO DATA IN REAL-TIME VIDEO CONFERENCE") provides an example of how to monitor bandwidth during an AV conference. Thus, each participant system such as Figure 3B participant systems 210, 214, and 216 in can monitor these states to determine if there are conditions requiring a fallback to the common codec. For example, if the network bandwidth (e.g., upload bandwidth) for the participant system degrades too much to support the transmission of multiple codec streams, or if the participant system has become too hot to support the transmission of multiple codec streams, or if the main battery has been depleted too low to support the transmission of multiple codec streams, then in Figure 3A operation 203 of, the participant system switches from transmitting a first stream encoded with a first (better) codec to transmitting a second stream encoded with the common codec; at some point during this switch, the participant system will stop transmitting the first stream so that it can transmit only the second stream to all receivers. In Figure 3A before operation 203 in, Figure 3BIn the example shown, the participant system 210 will have transmitted the second stream 219 to the participant system 214 (while also transmitting the first stream 217 to the participant system 216), so the handover can be performed by tagging the second stream 219 with the identifier of the first stream 217 or associating it with the identifier of that first stream, and this will cause the forwarding server 212 to forward the second stream to the participant system 216 (which had previously subscribed to the first stream). Tagging the second stream with the identifier of the first stream or associating it with the identifier of that first stream will automatically forward the second stream 219 to the participant system 216, and once the participant system 210 starts tagging the second stream 219 with the identifier of the first stream, it can stop transmitting the first stream 217. Such tagging or association can be referred to as a composite stream identifier because during fallback situations the common codec stream is associated with two stream identifiers. U.S. Patent No. 10,931,725 provides additional information regarding the use of composite stream identifiers for AV conferencing.

[0042] Figure 3A and Figure 3B The fallback method shown allows a participant system that transmits at least a common codec stream and a better codec stream to immediately switch to transmitting only the common codec stream in a fallback situation (e.g., network degradation) without the need for signaling or messages regarding the change. Thus, such participant systems can react very quickly to changing conditions when necessary. Once the conditions improve (e.g., network bandwidth improves), the participant system can resume transmitting both the common codec stream 219 and the first stream 217; each participant system can perform operation 201 after a fallback situation to determine whether it can resume the transmission of both streams. Resuming the transmission of both streams requires the participant system to start generating and transmitting the first stream (encoded with a better codec) and stop tagging the second stream (which is associated with the identifier of the first stream) with the identifier of the first stream.

[0043] Figure 4A and Figure 4B The fallback method shown uses messages or signaling to complete the fallback and does not use a composite stream identifier. In Figure 4A operation 251, each participant system monitors its local state; operation 251 can be similar to Figure 3A operation 201. When the monitoring shows that a fallback is required, the participant system (e.g., participant system 275) performs operation 253 by switching from transmitting both a better codec stream (the first stream) and the common codec stream to transmitting only the common codec stream. Figure 4A The method shown can operate in the context of the Figure 4B AV conference shown. The handover involves sending a message (e.g., Figure 4BThe shown Stop 283 message) is sent to a forwarding server (e.g., forwarding server 277), which in turn sends a message to a participant system (e.g., participant system 281) that has received the first stream 283 from the participant system 275 that sent the Stop 283 message, and the message from the forwarding server includes information that the first stream 283 will not continue to be transmitted from the participant system 275. Before operation 253, the participant system has created a common codec stream 285 via the forwarding server 277 and sent it to the participant system 279, and that participant system has transmitted a better codec stream 283 to the participant system 281. When a fallback condition occurs, the participant system 281 receives a message (e.g., data based on the Stop 283 message) and then subscribes to the common codec stream by sending a subscription message (e.g., subscribe to common) to the forwarding server 277. This subscription message causes the forwarding server 277 to start forwarding the common codec stream to the participant system 281. In one embodiment, once the Stop 283 message is transmitted to the forwarding server 277, the participant system 275 may stop creating and transmitting the better codec stream 283. The participant may continue to monitor its local state and implement such recovery when conditions permit resuming transmission of the better codec stream. In one embodiment, the recovery may involve the forwarding server publishing the availability of a new stream (better codec stream), which the forwarding server forwards the published availability to participant systems that can use the better codec stream; in turn, these participant systems may subscribe to the better codec stream again.

[0044] Another aspect of the present disclosure relates to using a preferred or highest quality codec at each participant system, which the participant system can utilize while using a common codec available at all participant systems in an AV conference. The common codec can be used as the primary codec for some participants who do not support a better codec (where the bitrate of the stream encoded with the common codec is varied to cope with network bandwidth changes) and can also be used as a fallback codec for those participant systems that can support a better codec. This aspect is shown in Figure 5A and Figure 5B which shows, Figure 5A a method of an embodiment of this aspect, Figure 5B shows an example of an AV conference environment in which the method shown in Figure 5A can be used. In operation 301 in Figure 5A , each participant system can exchange a list of its codec capabilities (and optionally other media-related capabilities) for the AV conference with all other participant systems (or at least one subset of all participant systems). For example, each participant system can send to a forwarding server (such as Figure 5BOne or more messages of the illustrated forwarding server 317) publish its capabilities list. The result of such an exchange can provide each participant with the available codecs at other participants and allow each participant to select a codec from the list of available codecs of other participants. This implements operation 303, in which each participant can subscribe to the preferred or highest quality codecs available to each participant (for those participants that more support the preferred or highest quality codecs rather than the common codec), while allowing such participant systems to fallback to using the common codec in a fallback situation. The fallback situation can use the fallback methods described above. In Figure 5B In the illustrated example, participant systems 310 and 312 each have a set of the same codec capabilities, which includes a common codec (Codec X). Participant systems 316 and 314 each have a set of the same codec capabilities, which includes a common codec. In this example, participant systems 310 and 312 can select a better codec such as Codec A or B when not in a fallback condition (e.g., when the network bandwidth is too low) to transmit and receive streams between these systems 310 and 312 in an AV conference, and these participant systems 310 and 312 can transmit and receive streams encoded with the common codec to and from participants 316 and 314. Participants 316 and 314 can transmit and receive common codec streams from all other participants. When there is a fallback condition (e.g., network bandwidth reduction or overheating of the thermal state or too low battery power), the common codec stream can be used as the fallback stream for participant systems 310 and 312.

[0045] The embodiments described herein can also use additional methods performed by a server system, such as a forwarding server, to match different participant systems. For example, in one embodiment, the server can manipulate, add, or remove media headers and control commands to match different participant systems with different media capabilities so that they can join and maintain an AV conference. For example, the server can assist different participant systems in terms of loss recovery, media synchronization, media attributes such as image / video orientation, and media attributes such as audio power (e.g., volume) levels, as well as transport headers and additional encryption (when needed).

[0046] Figure 6 An example of a data processing system 800 that can be used with one embodiment is shown. For example, system 800 can be implemented to provide a system or device that executes any of the methods described herein. For example, system 800 can be used to implement a participant system or a forwarding server. Note that although Figure 6Shows various components of the device, but it is not intended to represent any particular architecture or manner of interconnecting these components, so such details are not closely related to the present disclosure. It should also be understood that network computers and other data processing systems or other consumer electronic devices with fewer components or possibly more components can also be used in the embodiments of the present disclosure.

[0047] As Figure 6 shown, a device 800 in the form of a data processing system includes a bus 803 coupled to a microprocessor 805, a ROM (Read Only Memory) 807, a volatile RAM 809, and a non-volatile memory 811. The one or more microprocessors 805 can retrieve instructions from the memories 807, 809, 811 and execute the instructions to perform the above operations. The one or more microprocessors 805 can include one or more processing cores. The bus 803 interconnects these various components together and also interconnects these components 805, 807, 809, and 811 to a display controller and a display device 813, as well as to peripheral devices such as input / output (I / O) devices 815, which can be a touch screen, a mouse, a keyboard, a modem, a network interface, a printer, and other devices well known in the art. Generally, the input / output device 815 is coupled to the system through an input / output controller 810. The volatile RAM (Random Access Memory) 809 is typically implemented as a dynamic RAM (DRAM) that requires continuous power supply to refresh or maintain the data in the memory.

[0048] The non-volatile memory 811 is typically a magnetic hard disk drive, or a magneto-optical drive, or an optical drive, or a DVD RAM, or a flash memory, or other type of memory system that retains data (e.g., a large amount of data) even after the system is powered off. Generally, the non-volatile memory 811 can also be a random access memory, although this is not necessary. Although Figure 6 shown that the non-volatile memory 811 is a local device directly coupled to the remaining components in the data processing system, it should be understood that the embodiments of the present disclosure can utilize a non-volatile memory remote from the system, such as a network storage device coupled to the data processing system through a network interface such as a modem, an Ethernet interface, or a wireless network. The bus 803 can include one or more buses interconnected by various bridges, controllers, and / or adapters well known in the art.

[0049] Portions of the above-described subject matter can be implemented using logic circuitry such as dedicated logic circuits or using a microcontroller or other form of processing core that executes program code instructions. Thus, program code such as machine-executable instructions can be used to perform the processes taught by the above discussion, the machine-executable instructions causing the machine to execute the instructions to perform certain functions. In this context, a "machine" can be a machine that converts intermediate form (or "abstract") instructions into processor-specific instructions (e.g., an abstract execution environment such as a "virtual machine" (e.g., the Java virtual machine), an interpreter, the common language runtime, a high-level language virtual machine, etc.), and / or an electronic circuit disposed on a semiconductor chip (e.g., a "logic circuit" implemented using transistors) designed to execute instructions such as a general-purpose processor and / or a dedicated processor. The processes taught by the above discussion can also be performed by (in place of or in combination with a machine) an electronic circuit designed to perform the process (or a portion thereof) without executing program code.

[0050] The present disclosure also relates to an apparatus for performing the operations described herein. The apparatus can be specifically constructed for the required purposes or can include a general-purpose device selectively activated or reconfigured by a computer program stored in the device. Such a computer program can be stored in a non-transitory computer-readable storage medium such as, but not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magneto-optical disks, DRAM (volatile), flash memory, read-only memory (ROM), RAM, EPROM, EEPROM, magnetic or optical cards, or any type of medium suitable for storing electronic instructions, and each is coupled to the device bus.

[0051] A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). For example, a non-transitory machine-readable medium includes read-only memory ("ROM"); random access memory ("RAM"); magnetic disk storage media; optical storage media; flash devices; and the like.

[0052] An article of manufacture can be used to store program code. The article of manufacture storing the program code can be embodied as, but not limited to, one or more non-transitory memories (e.g., one or more flash memories, random access memories (static, dynamic, or otherwise)), optical disks, CD-ROMs, DVD ROMs, EPROMs, EEPROMs, magnetic or optical cards, or other types of machine-readable media suitable for storing electronic instructions. Program code can also be downloaded from a remote computer (e.g., a server) to a requesting computer (e.g., a client) via a data signal embodied in a propagated medium (e.g., via a communication link (e.g., a network connection)) and then stored in a non-transitory memory (e.g., DRAM or flash memory or both) of the client computer.

[0053] The foregoing detailed description has been presented in terms of algorithms and symbolic representations of operations on data bits within a device memory. These algorithmic descriptions and representations are the tools used by those skilled in the data processing arts, and they serve to most effectively convey the substance of their work to others skilled in those arts. An algorithm is here, and generally, a self-consistent sequence of operations leading to a desired result. These operations are those requiring physical manipulation of physical quantities. Usually, but not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of commonality, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, and the like.

[0054] However, it should be borne in mind that all of these and similar terms are to be associated with appropriate physical quantities and are merely convenient labels applied to those quantities. Unless specifically stated otherwise, it will be apparent from the foregoing discussion that, throughout the specification, discussions using terms such as "receiving," "determining," "transmitting," "terminating," "waiting," "changing," etc., refer to actions and processes of a device or similar electronic computing device that manipulates data represented as physical (electronic) quantities within the registers and memory of the device and transforms it into other data similarly represented as physical quantities within the memory or registers of the device or other such information storage, transmission, or display devices.

[0055] The processes and displays presented herein are not inherently related to any particular device or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus for performing the described operations. The required structure for various such systems will be apparent from the following description. In addition, the present disclosure has not been described with reference to any particular programming language. It should be understood that a variety of programming languages may be used to implement the teachings of the present disclosure as described herein.

[0056] In the foregoing specification, specific exemplary embodiments have been described. It will be apparent that various modifications may be made to those embodiments without departing from the broader spirit and scope given by the following claims. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a limiting sense.

Claims

1. A non-transitory machine-readable medium storing executable program instructions that, when executed by a first data processing system, cause the first data processing system to perform a method in an environment including the first data processing system, a second data processing system, a third data processing system, and a server, the method comprising: Transmitting, by the first data processing system, codec capabilities of the first data processing system to the server, the codec capabilities of the first data processing system indicating a first set of codecs available in the first data processing system for use in an audio-visual (AV) conference; Receiving, by the first data processing system, a second set of codec capabilities of the second data processing system, the second set of codec capabilities indicating a second set of codecs available in the second data processing system for use in the AV conference; Receiving, at the first data processing system, a request from the second data processing system to receive a first stream encoded with a first codec in the AV conference, the first codec being in the first set of codecs in the first data processing system; Transmitting, by the first data processing system, the first stream to the server for delivery to the second data processing system; Receiving, by the first data processing system, a request from the third data processing system for a second stream encoded with a second codec in the AV conference, the second codec being in the first set of codecs and different from the first codec; Creating, on demand and in response to the request from the third data processing system, the second stream in the AV conference; And Transmitting the second stream to the server for delivery to the third data processing system while the first data processing system continues to transmit the first stream to the server for delivery to the second data processing system.

2. The non-transitory machine-readable medium according to claim 1, wherein each codec in the first set of codecs and the second set of codecs is configured to compress video content in the AV conference for transmission to other participants in the AV conference and to decompress video content received in the AV conference.

3. The non-transitory machine-readable medium according to claim 1, wherein the second codec is common to the first data processing system, the second data processing system, and the third data processing system and is capable of being used in the first data processing system, the second data processing system, and the third data processing system.

4. The non-transitory machine-readable medium according to claim 1, wherein the first data processing system and the second data processing system execute one or more versions of a first operating system during the AV conference, and the third data processing system executes a second operating system during the AV conference.

5. The non-transitory machine-readable medium according to claim 1, wherein the method further comprises: Receiving, at the first data processing system, a third stream in the AV conference from the second data processing system, the third stream being encoded with the first codec; Decoding the third stream using the first codec; Receiving, at the first data processing system, a fourth stream in the AV conference from the third data processing system, the fourth stream being encoded with the second codec; Decoding the fourth stream using the second codec at the first data processing system; And Displaying, on a display of the first data processing system, video from the third stream, video from the fourth stream, and video from the first stream.

6. The non-transitory machine-readable medium according to claim 3, wherein the method further comprises: Monitoring, by the first data processing system, one or more of the following: uplink bandwidth from the first data processing system, thermal data regarding a thermal state of the first data processing system, or battery state regarding a battery in the first data processing system; In response to one or more conditions determined from the monitoring, switching from transmitting the first stream to the second data processing system to transmitting the second stream to the second data processing system.

7. The non-transitory machine-readable medium according to claim 6, wherein the switching comprises: Associating the stream identifier of the second stream with that of the first stream to cause the server to forward the second stream to the second data processing system.

8. The non-transitory machine-readable medium according to claim 6, wherein the method further comprises: Transmitting, by the first data processing system, an indication to the server that the first data processing system has or will stop transmitting the first stream to be delivered to the second data processing system, for delivery to the second data processing system; Receiving a request for the second stream from the second data processing system; Transmitting, by the first data processing system, the second stream to the server for delivery to the second data processing system.

9. A method performed by the server in an environment including a server, a first data processing system, a second data processing system, and a third data processing system, the method comprising: Receiving, by the server, codec capabilities of the first data processing system from the first data processing system, the codec capabilities of the first data processing system indicating a first set of codecs available in the first data processing system for use in an audio-visual (AV) conference; Transmitting, by the server, a second set of codec capabilities of the second data processing system to the first data processing system, the second set of codec capabilities indicating a second set of codecs available in the second data processing system for use in the AV conference; Transmitting, by the server, a request from the second data processing system to the first data processing system to receive a first stream encoded with a first codec in the AV conference, the first codec being in the first set of codecs in the first data processing system; Receiving, by the server, the first stream from the first data processing system for delivery to the second data processing system; The server transmits a request for a second stream encoded with a second codec in the AV conference from the third data processing system to the first data processing system, the second codec being in the first group of codecs and different from the first codec; and receiving, at the server, the second stream for delivery to the third data processing system while the server continues to transmit the first stream to the second data processing system.

10. The method according to claim 9, wherein each codec in the first group of codecs and the second group of codecs is configured to compress video content in an AV conference for transmission to other participants in the AV conference and is configured to decompress video content received in the AV conference, and wherein the server is a group of one or more data processing systems.

11. The method according to claim 10, wherein the second codec is a codec common to the first data processing system, the second data processing system, and the third data processing system and is capable of being used in the first data processing system, the second data processing system, and the third data processing system.

12. The method according to claim 11, wherein the method further comprises: switching from transmitting the first stream to the second data processing system to transmitting the second stream to the second data processing system.

13. The method according to claim 12, wherein the switching comprises: Receiving a stream identifier of the second stream to enable the server to forward the second stream to the second data processing system.

14. The method according to claim 12, wherein the method further comprises: receiving, by the server, from the first data processing system an indication that the first data processing system has or will stop transmitting the first stream to be delivered to the second data processing system for delivery to the second data processing system; transmitting a request for the second stream from the second data processing system; receiving, by the server, the second stream from the first data processing system for delivery to the second data processing system.

15. A method performed by the first data processing system in an environment including a first data processing system, a second data processing system, a third data processing system, and a server, the method comprising: transmitting, by the first data processing system, codec capabilities of the first data processing system to the server, the codec capabilities of the first data processing system indicating a first group of codecs available for use in an audiovisual AV conference in the first data processing system; receiving, by the first data processing system, second group codec capabilities of the second data processing system, the second group codec capabilities indicating a second group of codecs available for use in the AV conference in the second data processing system; receiving, at the first data processing system, a request from the second data processing system to receive a first stream encoded with a first codec in the first group of codecs in the first data processing system; The first data processing system transmits the first stream to the server for delivery to the second data processing system; The first data processing system receives a request from the third data processing system for a second stream encoded with a second codec in the AV conference, the second codec being in the first set of codecs and different from the first codec; On demand and in response to the request from the third data processing system, create the second stream in the AV conference; And Transmit the second stream to the server for delivery to the third data processing system, while the first data processing system continues to transmit the first stream to the server for delivery to the second data processing system.

16. The method according to claim 15, wherein each codec in the first set of codecs and the second set of codecs is configured to compress video content in an AV conference for transmission to other participants in the AV conference and is configured to decompress video content received in the AV conference.

17. The method according to claim 15, wherein the second codec is common to the first data processing system, the second data processing system, and the third data processing system and can be used in the first data processing system, the second data processing system, and the third data processing system.

18. The method according to claim 15, wherein the first data processing system and the second data processing system execute one or more versions of a first operating system during the AV conference, and the third data processing system executes a second operating system during the AV conference.

19. The method according to claim 15, wherein the method further comprises: Receiving, at the first data processing system, a third stream in the AV conference from the second data processing system, the third stream being encoded with the first codec; Decoding the third stream using the first codec; Receiving, at the first data processing system, a fourth stream in the AV conference from the third data processing system, the fourth stream being encoded with the second codec; Decoding the fourth stream using the second codec at the first data processing system; And Displaying, on a display of the first data processing system, video from the third stream, video from the fourth stream, and video from the first stream.

20. The method according to claim 17, wherein the method further comprises: Monitoring, by the first data processing system, one or more of the following: uplink bandwidth from the first data processing system, thermal data regarding a thermal state of the first data processing system, or battery state regarding a battery in the first data processing system; In response to one or more conditions determined from the monitoring, switching from transmitting the first stream to the second data processing system to transmitting the second stream to the second data processing system.

21. The method according to claim 20, wherein the switching comprises: Associate the second stream with the stream identifier of the first stream so that the server forwards the second stream to the second data processing system.

22. The method according to claim 20, wherein the method further comprises: Transmitting, by the first data processing system, an indication to the server that the first data processing system has or will stop transmitting the first stream to be delivered to the second data processing system, for delivery to the second data processing system; Receiving, from the second data processing system, a request for the second stream; Transmitting, by the first data processing system, the second stream to the server for delivery to the second data processing system.

23. A non-transitory machine-readable medium storing executable program instructions that, when executed by a first data processing system, cause the first data processing system to perform a method in an environment including the first data processing system, a second data processing system, and a server, the method comprising: Transmitting, by the first data processing system, the codec capabilities of the first data processing system to the server, the codec capabilities of the first data processing system indicating a first set of codecs available in the first data processing system for use in an audiovisual (AV) conference; Receiving, by the first data processing system, a second set of codec capabilities of the second data processing system, the second set of codec capabilities indicating a second set of codecs available in the second data processing system for use in the AV conference; Transmitting, by the first data processing system, a request to the server for a first stream encoded by a first codec at the second data processing system, the request based on a standard at the first data processing system for the highest quality codec common to the first set of codecs and the second set of codecs.

24. A method performed by a server in an environment including the server, a first data processing system, a second data processing system, and a third data processing system, the method comprising: Receiving, by the server, an identification of the codec capabilities of the first data processing system from the first data processing system, the identification indicating a first set of codecs available in the first data processing system for use in an audiovisual (AV) conference, each codec associated with a corresponding coding quality; Receiving, by the server, an identification of the codec capabilities of the second data processing system from the second data processing system, the identification indicating a second set of codecs available in the second data processing system for use in the AV conference, each codec associated with a corresponding coding quality; Selecting, based on quality selection, a first codec to be used in the AV conference from the first set of codecs and the second set of codecs; Transmitting the identification of the selected first codec to the first data processing system and the second data processing system; In response to a request to join the AV conference from the third data processing system: The server receives an indication of the codec capabilities of the third data processing system, the indication indicating a second set of codecs available in the second data processing system for use in the AV conference, each codec being associated with a corresponding coding quality; when the selected encoder is not included in the indication of the codec capabilities of the third data processing system, select a second codec from the indication of the codec capabilities of the third data processing system, and transmit the selection of the second codec to at least one of the first data processing system and the second data processing system and the third data processing system.

25. The method according to claim 24, further comprising: after transmitting the indication of the selected first codec to the first data processing system and the second data processing system, exchanging data of the AV conference encoded according to the selected first codec between the first data processing system and the second data processing system, and after transmitting the indication of the selected second codec to the third data processing system and the one of the first data processing system and the second data processing system, exchanging data of the AV conference encoded according to the selected second codec between the third data processing system and the one of the first data processing system and the second data processing system.

Images