System and method for displaying a teleconference session

Abstract

Systems and methods for displaying a teleconference session are described herein. Teleconference data including display streams is sent to a plurality of client computers. Each of the streams may include at least a video component and an audio component. A user interface includes a first display area presenting a first stream and a second display area presenting a second stream. An event may occur that removes the video component from the first stream. In response to the event, the presentation of the first stream may be removed from the first display area, and one or more of the plurality of streams may be displayed in at least a portion of the first display area. This technique of modifying the user interface display based on the action of a select user who has switched to an audio-only mode may optimize the use of the display area.

Description

[0001] This application is a divisional application of the invention patent application "System and method for displaying telephone conference conversations" with application date of March 7, 2018 and application number 201880018024.6. Technical Field

[0002] The present disclosure relates to systems and methods for teleconferencing. Background Art

[0003] The use of teleconference systems has been increased in business and corporate environments, promoting meetings and seminars between people in remote locations, reducing the need to travel so that these people are gathered together for meetings or seminars. Usually, teleconference systems allow users (that is, people) of teleconference systems in two or more remote locations to communicate interactively with each other through real-time, simultaneous two-way video streams, audio streams or both. Some teleconference systems (for example, Cisco WebEx provided by Cisco Systems, San Jose, California, GoToMeeting provided by Citrix Systems, Santa Clara, California, Zoom provided by Zoom Video Communications, San Jose, California, Google Hangouts provided by Alphabet, Mountain View, California, and Skype for Business provided by Microsoft, Redmond, Washington) also allow users to exchange digital documents such as images, texts, videos and any other forms.

[0004] Most teleconferencing systems utilize remote communication devices (e.g., video terminals, personal computers (desktop and portable), and mobile devices such as tablet devices, smartphones, etc.) to display video and play audio from a meeting or conference on a video display, the size of which may vary based on the type of communication device or video display being used. As a result, remote users of teleconferencing systems are typically limited to viewing the interactions of the meeting or conference through a "window" of the meeting or conference created by the video display, which may be the screen of a mobile device, a computer monitor, or a large video display.

[0005] This results in user interfaces that produce a fixed experience for remote users of a teleconferencing system participating in a meeting or conference. That is, some user interfaces only allow users to view framed individuals (i.e., images of other people participating in the conference) in a gallery experience, where a large amount of negative space in the user interface does not interact with these remote users. In addition, the individual framed images used by traditional user interfaces are not dynamically rendered, which increases the fixed experience of remote users participating in a meeting or conference. In addition, in some teleconferencing systems, remote users can see multiple displays of a fixed data stream on their video display, but there are few options to allow individual remote users to customize their views to view important or significant activities of the teleconferencing session. In addition, those teleconferencing systems do not dynamically customize views based on the content associated with the data stream.

[0006] Therefore, there is a need for an improved teleconferencing system that addresses these and other issues.With respect to these and other considerations, the disclosure herein is presented. Summary of the invention

[0007] The technology disclosed herein provides an event condition view for a teleconference session. The technology disclosed herein optimizes the use of a display screen during a teleconference. A participant can participate in a teleconference session and share a teleconference session feed including audio and video with others. When a participant stops sharing his or her video feed and transmits only an audio feed, the client that displays the video stream for the participant can automatically reconfigure the user interface to replace the presentation of the participant who stops sharing his or her video source with the stream of other participants. In some configurations, the presentation of the participant's teleconference session feed is replaced with one or more teleconference session feeds of people or content selected based on the activity level. In some configurations, the presentation of the participant's teleconference session feed is replaced only when the teleconference session feed includes at least the video feed or content of the participant who has been converted to audio-only sharing. In addition, in some configurations, the presentation of the participant's teleconference session feed is replaced only when the presentation of the teleconference session feed includes the video feed or content of the participant who has been converted to audio-only sharing, and the participant's teleconference feed is generated by the computing device used by the participant or originates from the computing device used by the participant. As will be described in more detail below, this feature is referred to as removal of the "ME" display. Embodiments utilizing these features replace the "ME" display with other content or people in the teleconference session.

[0008] In one illustrative example, a teleconference session can be displayed in a user interface arrangement on a client computing device. The user interface arrangement may include a first display area and a second display area. A first teleconference data stream can be displayed in the first display area, and a second teleconference data stream can be displayed in the second display area. An event may occur that removes a video component from the first stream of teleconference data. In response to the event, the first teleconference data stream is removed from the first display area, and then one or more streams are presented in the first display area. In some implementations, the first stream of teleconference data is provided by a client computing device displayed in the user interface. Thus, the described technology provides a user interface arrangement that dynamically arranges teleconference data streams based on one or more events. Dynamically arranging teleconference data streams based on one or more events can enhance the user experience of the teleconference session.

[0009] In one illustrative example, a teleconference session can be displayed in a user interface arrangement on a client computing device. The user interface arrangement can include a first display area presenting a first teleconference data stream and a second display area presenting a second teleconference data stream. A certain event can remove a video component from the first teleconference data stream. In response to the event, the first stream can be removed from the first display area. One or more streams including a video component and an audio component can replace the first stream removed from the first display area. In some implementations, the one or more streams including the video component can be presented in at least a portion of the first display area. The audio component associated with the one or more streams can be transmitted to an output device such as a speaker.

[0010] In some implementations, multiple streams including teleconference data can be sent to a client computing device. The client computing device can be caused to display a user interface arrangement including a first display area and a second display area. A first stream of the multiple streams can be presented in the first display area, and a second stream of the multiple streams can be presented in the second display area. A certain event can remove the video component from the first teleconference data stream. In response to the event, the first teleconference data stream can be subsequently removed from the first display area. In some implementations, in response to the event, the user interface arrangement can be rearranged so that the second stream is presented in at least a portion of the first display area. In addition, in some implementations, in response to the event, a third stream of the multiple streams can be presented in a third display area. The third stream can include an audio component or a video and an audio component. In addition, in some implementations, the first stream includes teleconference data generated by the client computing device.

[0011] In some embodiments, media streams are received from multiple client computing devices at a server. The server combines these media streams to generate teleconference data for defining various aspects of a teleconference session. Teleconference data may include separate data streams, also referred to herein as "streams," which may include content streams or participant streams. Participant streams include videos of one or more participants. Content streams may include videos or images of files, data structures, word processing documents, formatted documents (e.g., PDF documents), electronic forms, or presentations. Content streams include all streams that are not participant streams. In some configurations, participant streams may include video data, and in some configurations include audio data, which is streamed in real time from a video camera connected to a participant client computing device. In some instances, a participant may not have access to a camera, but a participant stream including an image of the participant, or an image representing the participant (e.g., an avatar) may be transmitted.

[0012] This summary is provided to introduce in simplified form a selection of concepts that are further described in the detailed description below. This summary is not intended to identify key features or essential features of the present invention, nor is it intended to be used to help determine the scope of protection of the present invention. For example, the term "technique" may refer to systems, methods, computer-readable instructions, modules, algorithms, hardware logic and / or operations as permitted in the context described above and throughout this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a block diagram of an example of a teleconferencing system.

[0014] Figure 2 yes Figure 1 A block diagram of an example of equipment in a teleconferencing system.

[0015] Figure 3A and 3B is a screenshot overlay view of a display corresponding to one of the client computing devices in a teleconference session showing a user interface arrangement.

[0016] Figure 3C and 3D is a screenshot overlay view of a display corresponding to one of the client computing devices in a teleconference session showing a user interface arrangement.

[0017] Figures 3E to 3I is a screenshot overlay view of a display corresponding to one of the client computing devices in a teleconference session showing a user interface arrangement.

[0018] Figure 4A and 4Bis a screen shot persistent view of a display corresponding to one of the client computing devices in the teleconference session showing a user interface arrangement.

[0019] Figure 4C and 4D is a screen shot persistent view of a display corresponding to one of the client computing devices in the teleconference session showing a user interface arrangement.

[0020] FIG. 5A to FIG. 5H is a screen shot view of a display corresponding to one of the client computing devices in a teleconference session showing a user interface arrangement with a drill-in feature.

[0021] Figure 6 is shown for Figure 1 A flowchart of the operations of providing a user interface arrangement in an exemplary teleconferencing system. DETAILED DESCRIPTION

[0022] The example described below enables the system to provide a teleconference session at a client computing device, wherein the client computing device has the ability to make event-driven adjustments to the session displayed on the client computing device. The teleconference session can be an event adjusted at a teleconference server, which is connected to multiple client computing devices participating in the teleconference session. The client computing device can be configured to allow the user to use the graphical user interface controls available to the user to adjust his or her viewing experience of the teleconference session during the session. In an exemplary implementation, the teleconference session receives a participant stream from a client computing device used by a participant. The participant stream includes video, audio, or image data of the participant to identify or represent the participant in the display of the teleconference session at the client computing device. The teleconference session can also receive a content stream from one or more client computing devices or from another source. The content stream includes all streams that are not participant streams. In some configurations, the content stream includes video or image data of a file, a data structure, a word processing document, a formatted document (e.g., a PDF document), a spreadsheet, or a presentation to be presented to the participant in the display of the teleconference and thereby shared with it. The teleconference session at the server combines the streams to generate teleconference data and sends the teleconference data to each client computing device according to the teleconference view configured for each client computing device.

[0023] In some implementations, a teleconference session includes an event-driven display of teleconference data. For example, a teleconference session can be displayed in a user interface arrangement on a client computing device. The user interface arrangement may include a first display area and a second display area. A first teleconference data stream can be displayed in the first display area, and a second teleconference data stream can be displayed in the second display area. An event may occur that removes a video component from the first teleconference data stream. In response to the event, the first teleconference data stream is removed from the first display area, and then one or more streams are presented in the first display area. The one or more streams presented in the first display area may include an audio component or an audio and video component. In some implementations, the first teleconference data stream is provided or generated by a client computing device that displays the user interface. Specifically, when an event removes the video component associated with the first stream, some implementations remove the first stream from the user interface arrangement.

[0024] Thus, some described implementations provide a user interface arrangement with dynamically removed and / or rearranged teleconference data streams based on one or more events. Dynamically arranging teleconference data streams based on one or more events can enhance the user experience of a teleconference session.

[0025] In the following description, various streams may be referred to as "streams" or "media streams". Content streams may be referred to as "content media streams" and participant streams may be referred to as "user media streams". As described above, user media streams (such as participant streams) may include audio and / or video or image data depicting or representing participants in a teleconference session. Content media streams (such as content streams) may be images or videos representing documents, videos, audio, or other media content that can be shared with a user of a client computing device for discussion between participants in a teleconference session.

[0026] The teleconference session view can be customized for each client computing device using one of several user interface arrangements. For a given client computing device, the teleconference session view can be in a first user interface arrangement referred to herein as an overlay view, a second user interface arrangement mentioned herein can be described as a persistent view, or a third or other user interface arrangement can describe other desired views. The overlay view provides an overall display experience in which people or content are viewed on a "stage" that is the main stream display area of ​​the interface. In some configurations, the main stream display area of ​​the interface can be displayed in a manner that dominates the display on the user's client computing device. The overlay view allows the user to participate in content shared between teleconference participants, or interact with the participants, or at least the most active participants.

[0027] In persistent view, content or participants are displayed in a primary stream display area, where the primary stream display area occupies a majority of the display, leaving room for a secondary stream display area. In persistent view, the secondary stream display area does not overlap the primary stream display area; such an arrangement is described herein as a side-by-side configuration. The secondary stream display area includes user interface elements for displaying participants or content that are not displayed in the primary stream display area.

[0028] exist Figure 1 , a diagram is shown for depicting an example of a teleconferencing system 100 in which system 102 can control the display of a teleconference session 104 according to an exemplary implementation. In this example, teleconference session 104 is between multiple client computing devices 106(1) to 106(N) (where N is a positive integer with a value of 2 or greater). Client computing devices 106(1) to 106(N) enable users to participate in teleconference session 104. In this example, teleconference session 104 can be hosted by system 102 over one or more networks 108. That is, system 102 can provide services that enable users of client computing devices 106(1) to 106(N) to participate in teleconference session 104. Alternatively, teleconference session 104 can be hosted by one of client computing devices 106(1) to 106(N) using peer-to-peer technology.

[0029] System 102 includes device 110, and device 110 and / or other components of system 102 may include distributed computing resources that communicate with each other, system 102, and / or with client computing devices 106(1) to 106(N) via one or more networks 108. In some examples, system 102 may be a stand-alone system that is tasked with managing various aspects of one or more teleconference sessions 104. For example, system 102 may be comprised of a server, such as a server, a service provider, or a server. GOOGLE Etc.

[0030] For example, the network 108 may include a public network such as the Internet, a private network such as an institutional and / or personal intranet, or some combination of private and public networks. The network 108 may also include any type of wired and / or wireless network, including but not limited to a local area network ("LAN"), a wide area network ("WAN"), a satellite network, a wired network, a Wi-Fi network, a WiMax network, a mobile communication network (e.g., 3G, 4G, etc.), or any combination thereof. The network 108 may utilize communication protocols including: packet-based and / or datagram-based protocols, such as Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), or other types of protocols. In addition, the network 108 may also include multiple devices that facilitate network communications and / or form the hardware foundation of the network, such as switches, routers, gateways, access points, firewalls, base stations, repeaters, backbone devices, and the like.

[0031] In some examples, network 108 may also include devices capable of connecting to a wireless network, such as a wireless access point (WAP). Examples support connection via WAPs that send and receive data over various electromagnetic frequencies (e.g., radio frequencies), including WAPs that support the Institute of Electrical and Electronics Engineers ("IEEE") 802.11 standards (e.g., 802.11g, 802.11n, etc.) and other standards.

[0032] In various examples, equipment 110 may include one or more computing devices operating in a cluster or other grouping configuration to share resources, balance loads, improve performance, provide failover support or redundancy, or for other purposes. For example, equipment 110 may belong to various types of equipment, such as traditional server type equipment, desktop computer type equipment and / or mobile type equipment. Therefore, although shown as a single type of equipment (server type equipment), equipment 110 may include various types of equipment, and is not limited to a particular type of equipment. Equipment 110 may represent but is not limited to a server computer, a desktop computer, a network server computer, a personal computer, a mobile computer, a laptop computer, a mobile phone, a tablet computer, or any other type of computing equipment.

[0033] The client computing device (e.g., one of client computing devices 106(1)-106(N)) may be of various types of devices, which may be the same as or different from device 110, such as a traditional client type device, a desktop type device, a mobile type device, a dedicated type device, an embedded device, and / or a wearable type device. Thus, the client computing device may include, but is not limited to, a desktop computer, a game console and / or a gaming device, a tablet computer, a personal digital assistant (“PDA”), a mobile phone / tablet device hybrid, a laptop computer, a telecommunications device, a computer navigation type client computing device (e.g., a satellite-based navigation system including a Global Positioning System (GPS) device), a wearable device, a virtual reality (“VR”) device, an augmented reality (“AR”) device, an implanted computing device, an automotive computer, a network-enabled television, a thin client, a terminal, an Internet of Things (“IoT”) device, a workstation, a media player, a personal video recorder (“PVR”), a set-top box, a camera, an integrated component (e.g., a peripheral device) for inclusion in a computing device, an appliance, or any other kind of computing device. In some implementations, the client computing device includes an input / output ("I / O") interface that enables communication with input / output devices such as: user input devices including peripheral input devices (e.g., game controllers, keyboards, mice, pens, voice input devices, touch input devices, gesture input devices, etc.) and / or output devices including peripheral output devices (e.g., displays, printers, audio speakers, tactile output devices, etc.).

[0034] The various classes and device types of client computing devices 106(1) through 106(N) may represent any type of computing device having one or more processing units 112 operably connected (e.g., via bus 116) to computer-readable media 114, which in some instances may include one or more of a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and any class of local, peripheral, and / or independent buses. The computer-readable media 114 may store executable instructions and data used by programmed functions during operation. Examples of functions implemented by executable instructions stored on the computer-readable media 114 may include, for example, an operating system 128, a client module 130, a profile module 132, and other modules, programs, or applications that may be loaded and executed by the processing unit 112.

[0035] The client computing devices 106(1)-106(N) may also include one or more interfaces 134 to enable communication with other input devices 148, such as a network interface, a camera, a keyboard, a touch screen 106(3), and a pointing device (mouse). For example, the interfaces 134 enable communication between the client computing devices 106(1)-106(N) and other networked devices (e.g., devices 110 and / or devices of system 102) via a network 108. Such network interfaces 134 may include one or more network interface controllers (NICs) or other types of transceiver devices to send and receive communications and / or data over a network.

[0036] exist Figure 1 In the example environment 100 of FIG. 1 , client computing devices 106(1) through 106(N) can use their respective client modules 130 to connect to each other and / or to other external devices in order to participate in a teleconference session 104. For example, a first user can utilize client computing device 106(1) to communicate with a second user of another client computing device 106(2). When executing client modules 130, the users can share data, which can cause client computing device 106(1) to connect to system 102 and other client computing devices 106(2) through 106(N) via network 108.

[0037] The client module 130 of each client computing device 106(1) to 106(N) may include logic for detecting user input and transmitting a control signal to the server to request a change in the view of the teleconference session 104 on the display. For example, Figure 1 The client module 130 in the first client computing device 106(1) can detect user input at the input device 148. For example, the user input can be sensed as a finger press on a user interface element displayed on a touch screen or a mouse click on a user interface element selected by a pointer on the display 150. The client module 130 converts the user input according to the function associated with the selected user interface element. The client module 130 can send a control signal 156(1) to a server (e.g., a server operating on the device 110) to perform the desired function.

[0038] In one exemplary function, a user of client computing device 106(1) may wish to switch a view on display 150 from a first user interface arrangement that provides a first look, feel, and experience for participants to a second user interface arrangement that provides a different look, feel, and experience. The user may, for example, click on a desired user interface element (e.g., a button) on the user's display 150. Client module 130 may recognize the click of the button as a request to switch views and send a corresponding control signal 156(1) to the conference call host to perform the view switching function. Control signal 156(1) in this function is a view switching control signal.

[0039] Client computing devices 106(1)...106(N) can use their respective profile modules 132 to generate participant profiles and provide the participant profiles to other client computing devices and / or devices 110 of system 102. The participant profile can include one or more of the following: an identification of the participant (e.g., name, unique identifier ("ID"), etc.); participant data, such as personal data and location data. The participant profile can be used to register participants for the teleconference session.

[0040] like Figure 1 As shown in , the device 110 of the system 102 includes a server module 136, a data store 138, and an output module 140. The server module 136 is configured to receive streams 142(1) to 142(M) (where M is a positive integer equal to 2 or greater) from each client computing device 106(1) to 106(N). In some scenarios, not all client computing devices participating in the teleconference session 104 provide instances of the stream 142, and therefore, M (the number of submitted instances) may not be equal to N (the number of client computing devices). In some other scenarios, one or more of the client computing devices may be transmitting other streams or media streams that include content such as documents or other similar types of media intended to be shared during the teleconference session.

[0041] The server module 136 is also configured to receive, generate, and transmit session data 144, and store the session data 144 in the data store 138. In various examples, the server module 136 can select aspects of the streams 142 to be shared with the client computing devices 106(1) to 106(N). The server module 136 can combine the streams 142 to generate teleconference data 146 that defines various aspects of the teleconference session 104. The teleconference data 146 can include various streams that include the selected streams 142. The teleconference data 146 can define various aspects of the teleconference session 104, such as the user interface layout of the user interface on the client, the type of data displayed, and other functions of the server and the client. The server module 136 can configure the teleconference data 146 for each client computing device 106(1)-106(N). The teleconference data can be divided into various instances referred to as 146(1)-146(N). Output module 140 may transmit teleconference data instances 146(1)-146(N) to client computing devices 106(1) through 106(N). Specifically, in this example, output module 140 transmits teleconference data 146(1) to client computing device 106(1), transmits teleconference data 146(2) to client computing device 106(2), transmits teleconference data 146(3) to client computing device 106(3), and transmits teleconference data 146(N) to client computing device 106(N), respectively.

[0042] Teleconference data instances 146(1)-146(N) may convey audio components that may include video components representing contributions from each participant in teleconference session 104. Each teleconference data instance 146(1)-146(N) may also be configured in a manner unique to the needs of each participant user of client computing devices 106(1)-106(N). Each client computing device 106(1)-106(N) may be associated with a teleconference session.

[0043] exist Figure 2, a system block diagram is shown that depicts components of an exemplary device 200 configured to provide a teleconference session 104 between client computing devices (e.g., client computing devices 106(1)-106(N)) according to an exemplary implementation. Device 200 may represent one of devices 110 and / or 106, where device 200 includes one or more processing units 202, computer-readable media 204, and communication interface 206. The components of device 200 may be operatively connected, for example, via bus 207, where bus 207 may include one or more of a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and any type of local, peripheral, and / or independent bus.

[0044] As used herein, a processing unit (e.g., processing unit 202 and / or processing unit 112) may represent, for example, a CPU-type processing unit, a GPU-type processing unit, a field programmable gate array ("FPGA"), another type of digital signal processor ("DSP"), or other hardware logic components that may be driven by a CPU in some instances. By way of example and not limitation, exemplary types of hardware logic components that may be used include application specific integrated circuits ("ASICs"), application specific standard products ("ASSPs"), systems on chips ("SOCs"), complex programmable logic devices ("CPLDs"), and the like.

[0045] As used herein, a computer-readable medium (e.g., computer-readable medium 204 and / or computer-readable medium 114) can store instructions that can be executed by a processing unit. The computer-readable medium can also store instructions that can be executed by an external processing unit (e.g., an external CPU, an external GPU) and / or can be executed by an external accelerator (e.g., an FPGA-type accelerator, a DSP-type accelerator, or any other internal or external accelerator). In various examples, at least one CPU, GPU, and / or accelerator is incorporated into the computing device, while in some examples, one or more of the CPU, GPU, and / or accelerator is external to the computing device.

[0046] Computer readable media may include computer storage media and / or communication media. Computer storage media may include one or more of the following: volatile memory, non-volatile memory, and / or other persistent and / or secondary computer storage media, removable and non-removable computer storage media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Thus, computer storage media includes media in tangible and / or physical form contained in a device and / or hardware component, which hardware component is part of the device or external to the device, including but not limited to random access memory ("RAM"), static random access memory ("SRAM"), dynamic random access memory ("DRAM"), phase change memory ("PCM"), read-only memory ("ROM"), erasable programmable read-only memory ("EPROM"), electrically erasable programmable read-only memory ("EEPROM"), flash memory, compact disc read-only memory ("CD-ROM"), digital versatile disc ("DVD"), optical cards or other optical storage media, cassettes, magnetic tape, magnetic disk storage, magnetic cards or other magnetic storage devices or media, solid-state storage devices, storage arrays, network attached storage, storage area networks, hosted computer storage, or any other storage memory, storage devices, and / or storage media that can be used to store and maintain information for access by a computing device.

[0047] In contrast to computer storage media, communication media may embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism. As defined herein, computer storage media does not include communication media. That is, computer storage media itself does not include communication media consisting solely of a modulated data signal, carrier wave, or propagation signal.

[0048] The communication interface 206 may represent, for example, a network interface controller (NICS) or other type of transceiver device to send and receive communications over a network. In some implementations, the communication interface 206 is used to communicate with the client computing device 106 over a data network.

[0049] In the example shown, computer-readable medium 204 includes data store 138. In some examples, data store 138 includes a data store such as a database, a data warehouse, or other type of structured or unstructured data store. In some examples, data store 138 includes a corpus and / or a relational database having one or more tables, indexes, stored procedures, etc. to enable data access including, for example, one or more of a Hypertext Markup Language (HTML) table, a Resource Description Framework (“RDF”) table, a Web Ontology Language (“OWL”) table, and / or an Extensible Markup Language (“XML”) table.

[0050] The data store 138 may store data for the operation of the processes, applications, components, and / or modules stored in the computer-readable medium 204 and / or data executed by the processing unit 202 and / or accelerator. For example, in some examples, the data store 138 may store session data 208 (e.g., session data 144), profile data 210, and / or other data. The session data 208 may include the total number of participants in the teleconference session 104, as well as the activities (e.g., the participants' behaviors, activities) that occurred in the teleconference session 104, and / or other data related to when and how the teleconference session 104 was conducted or hosted. Examples of the profile data 210 include, but are not limited to, participant identification ("ID") and other data.

[0051] In an exemplary implementation, the data store 138 stores data related to the view experienced by each participant on the display of the participant's client computing device 106. Figure 2 As shown in , data store 138 may include teleconference session views 250(1) through 250(N) corresponding to the display of each client computing device 106(1) through 106(N) participating in teleconference session 104. In this manner, system 102 may support individual control of the view experienced by each user during teleconference session 104. For example, as described below with reference to FIG. 3A to FIG. 3F and FIG. 4A to FIG. 4D As described in greater detail, system 102 allows a user to engage with an overlay view or persistent view displayed on a user client computing device 106. The overlay view is characterized by displaying desired media on a stage that may cover substantially the entire display screen. Controls, user interface elements such as icons, buttons, menus, and the like, and other elements not directly related to the presentation provided by the teleconference session on the display, simply do not appear. The persistent view provides a stage or primary stream display area that covers the primary area of ​​the display, and one or more edge or secondary stream display areas that present other aspects of the teleconference session.

[0052] In some implementations, the persistent view and the overlay view provide display areas, each of which includes a presentation of a teleconference data stream. The stream presented in each display area can be controlled by one or more events. For example, in one implementation, the first stream and the first display area include video and audio components. The user can toggle a user interface control (e.g., event) that removes the video component from the first display area. In response, the first stream can be removed from the first display area and replaced with another stream including video and audio components. In some implementations, the first stream is generated by a client computing device that displays a teleconference session. The conversion is transmitted to the server module 136 using a control signal 156, which is a conversion control signal for a conversion function. The server module 136 can modify the teleconference session view 250 accordingly. In some implementations, an event to remove the video component from the first display area is initiated from the teleconference system 100. For example, one or more devices associated with the teleconference system 100 can initiate the removal of the video from the first display area when it is determined that at least the communication channel for transmitting the video no longer supports or includes video data. This activity is referred to as an "audio-only" stream, or an event involving a switch to an audio-only stream. In another example, in response to switching to an audio-only stream, one or more devices associated with the teleconferencing system 100 may initiate removal of the video from the first display area, wherein the removal is based on one or more settings.

[0053] Persistent views and overlay views provide users with options for how the user experiences the teleconference session. The view on the user's display can be changed under user control to emphasize different aspects of the teleconference session based on the user's interests. A user may want to view content such as a document or presentation more closely than the media transmitted from the user. Each user can focus or promote content on a participant by having the content presented in the primary stream display area. Conversely, a user may want to engage more closely with a participant by promoting the media sent by the participant to the primary stream display area of ​​the display.

[0054] The teleconference session views 250(1)-250(N) may store data identifying the view being displayed for each client computing device 106(1)-106(N). The teleconference session views 250 may also store data related to the following streams: the media streams configured for display, the participants associated with those streams, whether the content media is part of the display, and information related to the content. Some teleconferences may involve a large number of participants. However, only a core number of participants can be referred to as "active participants." Each user's teleconference session view may be configured to focus on the media provided by the most active participants. Some teleconference sessions may involve a presenter (e.g., in a seminar or by one or more people giving a presentation). At any given time, one participant may be the presenter, and the presenter may occupy an enhanced role in the teleconference. The role of the presenter may be enhanced by maintaining a consistent presence on the user's display. Information related to the presenter may be maintained in the teleconference session view 250.

[0055] During a conference call, a participant may wish to switch to a different view of the conversation at any given time to emphasize or promote content or people in the user's view. The user may toggle the user interface element that represents the person or content that the user wants to promote. The toggle is communicated to the server module 136 using the control signal 156, which is a toggle control signal for the toggle function. The server module 136 may modify the conference call conversation view 250 accordingly.

[0056] As described above, the data store 138 may store profile data 210, streams 142, teleconference session views 250, session data 208, conversion functions 258, and switching functions 260. Alternatively, some or all of the above referenced data may be stored on a separate memory 224 on one or more processing units 202, such as a memory on a CPU-type processor, a GPU-type processor, an FPGA-type accelerator, a DSP-type accelerator, and / or another accelerator. In this example, the computer-readable medium 204 also includes an operating system 226 and an application programming interface 228, which is configured to expose the functions and data of the device 110 (e.g., the example device 200) to external devices associated with the client computing devices 106(1) to 106(N). In addition, the computer-readable medium 204 includes one or more modules (e.g., the server module 136 and the output module 140), but the number of modules shown is only an example and the number may be higher or lower. That is, the functions described herein in conjunction with the modules shown may be performed by a smaller number of modules or a larger number of modules on a single device or distributed across multiple devices.

[0057] As such, and as previously described, the system 102 is generally configured to host a teleconference session 104 with a plurality of client computing devices 106(1) to 106(N). The system 102 includes one or more processing units 202 and a computer-readable medium 204 having computer-executable instructions encoded thereon to cause the one or more processing units 202 to receive streams 142(1)-142(M) at the system 102 from the plurality of client computing devices 106(1)-106(N), select streams 142 based at least in part on each user's teleconference view 250, and transmit teleconference data 146 defining the teleconference session view 250 corresponding to the client computing devices 106(1) to 106(N). The teleconference data instances 146(1)-146(N) are transmitted from the system 102 to the plurality of client computing devices 106(1) to 106(N). The teleconference session views 250(1)-250(N) enable the plurality of client computing devices 106(1)-106(N) to display views of the teleconference session 104 under user control. The computer executable instructions may also cause the one or more processing units 202 to determine that the teleconference session 104 is to be transitioned to a different teleconference session view of the teleconference session 104 based on a view switching control signal 156 transmitted by the user.

[0058] In some implementations, the techniques disclosed herein may utilize one or more interface arrangements, also referred to as “views” or “teleconference session views.” In an example of operation, system 102 performs a method comprising: at system 102, receiving streams 142(1)-142N from multiple client computing devices 106(1)-106(N). The system combines and formats streams 142 based at least in part on a selected teleconference view of each client computing device to generate teleconference data 146 (e.g., teleconference data instances 146(1)-146(N)). The teleconference data instances 146(1)-146(N) are then transmitted to each client computing device 106(1)-106(N).

[0059] As will be described below, the interface arrangement of the teleconference session view may include a primary stream display area (referred to herein as a "canvas" or "stage") and a secondary stream display area (referred to herein as a display area of ​​a display for displaying "thumbnails"). In some implementations, the secondary stream display area may be separately configured as an area including selectable user interface elements for selecting streams of media streams associated with respective display areas. The predetermined interface arrangement may also include graphical elements ("control elements") that provide control functionality for the teleconference session. For example, graphical elements may be generated on a user interface that enable a user to provide content, end a session, mute one or more sounds, and the like.

[0060] In one illustrative example, the techniques disclosed below can utilize an overlay view. The overlay view can include one or more control elements. In some configurations, the control elements can be displayed to the user for a predetermined period of time and then faded out after the predetermined period of time has passed. In response to certain types of user activity (e.g., hovering a pointing device over a predetermined area of ​​a user interface), these control elements can reappear for a predetermined period of time.

[0061] The overlay view may include presentation of a person or content in the primary stream display area and presentation of a person or content in the auxiliary stream display area. The auxiliary stream display area may include user interface elements that may be presented for a predetermined period of time. These user interface elements are presented as an overlay on the primary stream display area when the user turns them on, and then fade out (e.g., are configured to disappear) after the predetermined period of time has passed.

[0062] The user can select a given user interface element in the secondary stream display area to "pin" the display of a particular content or person to the user interface. The user interface element of the content or user for which the "pinned" element is selected does not fade out over time, but remains displayed in the secondary stream display area overlaid on the primary stream display area.

[0063] In another illustrative example, the technology disclosed herein can utilize a persistent view. A persistent view can provide one or more control elements that can be configured. Typically, a persistent view is designed to provide an optimized interface for sharing content. A persistent view can include a presentation of a person or content in a primary stream display area, or content or a person in an auxiliary stream display area. In a persistent view, the presentation of a person or content in an auxiliary stream display area does not disappear after a predetermined period of time.

[0064] In another illustrative example, a primary stream display area may include multiple streams of teleconference data presented therein. The multiple streams may include video and audio components or only audio components. One of the multiple streams may include teleconference data generated by a computing device displaying the primary stream display area. The primary stream display area may be part of a user interface arrangement. The one of the multiple streams including teleconference data generated by the computing device may include video and audio components. An event such as a user-induced event or a system-level induced event may remove the video component from the one of the multiple streams generated by the computing device. In response to the event, the one of the multiple streams generated by the computing device may be removed from the primary stream display area. After removing the one of the multiple streams generated by the computing device from the primary stream display area, one or more of the remaining multiple streams of teleconference data presented in the stream main display area may be rearranged. In some implementations, another stream including video and audio components replaces one or more of the multiple streams removed. Similarly, the auxiliary stream display area may also include multiple streams of teleconference data presented therein. However, in some implementations, the presentation in the auxiliary stream display area may be smaller than the presentation in the primary stream display area.

[0065] It should be noted that the hosting of the teleconference session 104 by the system 102 described above enables control of the view of the teleconference session in the server function of the device 110. In some implementations, the server function of the device 110 can combine all media portions into the teleconference data of each client computing device 106 to configure the view to be displayed. The information stored in the teleconference session view as described above can also be stored in the data store of the client computing device. The client computing device can receive user input and convert the user input into a view switching control signal that is not sent to the server. The view switching control signal can be processed on the client computing device itself to cause the display to switch to the desired view. The client computing device 106 can change the display by reorganizing the media streams and content media streams (if any) in the teleconference data 146 received from the server according to the view selected by the user.

[0066] Reference now FIG. 3A to FIG. 3F, which illustrate exemplary displays in an overlay view. In the overlay view, the display 150 includes a primary stream display area 302, which is also referred to herein as the "stage." In the persistent view, the display 150 includes a primary stream display area 302 ("stage") and a secondary stream display area (which displays thumbnails of other streams). In some configurations, when the streams include a content stream, the system 100 can default to the persistent view. The user can select a user interface element, or provide any other command, to switch between the persistent view and the overlay view.

[0067] FIG. 3A to FIG. 3B An example of a display 150(1) is depicted, which is shown connected to Figure 1 1 , according to an exemplary implementation, displays an overlay view of the teleconference session 104 in a user OFF state.

[0068] The overlay view of display 150(1) includes a primary stream display area 302 extending over the screen area of ​​display 150(1). In some configurations, primary stream display area 302 is configured in a manner that dominates the display. In some configurations, primary stream display area 302 (e.g., the "stage" for the teleconference session) can be substantially edge-to-edge or full screen. As outlined above, when the user transitions to the persistent view, the interface can include a secondary stream display area (see FIG. 4A to FIG. 4D ). In some configurations, the primary stream display area 302 may be larger than the secondary stream display area. FIG. 3A to FIG. 3B The example of shows an example of an overlay view without displaying the auxiliary stream display area. A pointer 310 is provided to allow the user to interact with the view. In this example, the primary stream display area 302 displays two presentations, each of which is referred to herein as a graphical element 304a and 304b, each presentation corresponding to a user media stream. These streams may include an audio component, an audio and video component, or an audio component, and an image (e.g., an avatar or picture) representation of the user participating in the teleconference session. Figure 3F An exemplary presentation of an audio component and an image (e.g., an avatar or picture) is shown in FIG. 304. See in particular graphical element 304c. An exemplary presentation of a user media stream including only an audio component would be similar to graphical element 304c, but without the avatar or picture.

[0069] Two graphical elements 304a-304b are shown occupying Figure 3AThe user interface in the illustrated example is arranged with a main stream display area 302; however, any number of graphical elements may be displayed. In some examples, the number of graphical elements displayed may be limited to a maximum value for available bandwidth or as desired to limit the video cluster on display 150(1). In some implementations, when fewer than two participants are involved in a teleconference session, fewer than two graphical elements 304a-304b may be displayed. In a teleconference session involving more than the maximum number of graphical elements, the displayed graphical elements 304a-b may correspond to a dominant person or something that is considered an "active participant." The designation of "active participant" may be predefined as a reference to a particular person, or in some implementations, functionality may be provided to identify "active participants" versus passive participants by applying activity level priorities.

[0070] Activity level priorities rank participants based on their potential contribution to the conference call. In an exemplary implementation, activity level priorities for identifying active versus passive participants may be determined by analyzing each participant's media stream at server 136. The conference call system may include functionality for comparing the activities of the participants and dynamically promoting those who speak more frequently or move and / or speak more frequently as active participants.

[0071] The placement of the graphical elements 304a-304b may also reflect the activity level priority of the participants to which the graphical elements correspond. For example, the overlay view may be defined to have a convention where the top of the main stream display area 302 displays the graphical element 304b corresponding to the most prominent participant. In some sessions, the dominant participant may be the host.

[0072] In some implementations, the graphical element 304a is presented in the first display area 306. The first display area 306 is illustrated by a dashed box. Additionally, the graphical element 304b is presented in the second display area 308. The second display area 308 is also illustrated by a dashed box. The dashed box does not appear on the display 150(1).

[0073] In some implementations, graphical element 304a includes a video component and an audio component. An event may occur that causes the video component associated with graphical element 304a to be removed. The event may be a user-induced event, such as a user-toggled control associated with a user interface of client computing device 106(1), an event caused by a client computing device (e.g., client computing device 106(1)), or an event caused by device 110. In one example, client computing device 106(1) or system 100 may trigger an event that causes the video component associated with graphical element 304a to be removed when it is determined that the communication bandwidth does not provide reliable or user-consumable video.

[0074] System 100 may remove graphical element 304a from primary stream display area 302. Specifically, client computing device 106(1) and / or device 110 may remove graphical element 304a from primary stream display area 302 in response to removing the video component associated with graphical element 304a.

[0075] In some implementations, once the graphical element 304a is removed, the primary stream display area 302 is rearranged to present one or more teleconference data streams in the first display area 306. For example, the graphical element 304b may be enlarged (see Figure 3B ), so that the graphic element 304b is at least partially presented in the first display area 306.

[0076] In another example, the graphical element 304b remains in the second display area 308, and one or more teleconference data streams are presented in the first display area 306 at the location of the removed graphical element 304a. In a specific example, after the graphical element 304a is removed, the one or more teleconference data streams presented in the first display area 306 include video and audio components. In another example, after the graphical element 304a is removed, the one or more teleconference data streams presented in the first display area 306 include an audio component and an avatar or image component. Figure 3F An exemplary presentation of an audio component and an image (e.g., an avatar or picture) is shown in . See in particular graphical element 304c. An exemplary presentation of a user media stream including only an audio component is similar to graphical element 304c, but without the avatar or picture. Figure 3F An exemplary presentation of an audio component and an image (e.g., an avatar or picture) is shown in . See in particular graphical element 304c. In such a configuration, graphical element 375 (e.g., a ring around an avatar) can change display properties, such as being highlighted, changing thickness, or otherwise changing its appearance, to indicate when a user is speaking.

[0077] In some implementations, the graphical element 304a removed from the first display area 306 is associated with a teleconference data stream generated or initiated by a client computing device 106(1) associated with the display 150(1), also referred to herein as an "ME" user interface element or graphical element. In some cases, embodiments may remove the presentation of the "ME" user interface element or graphical element while not removing the presentation of other streams that are transitioned to an audio-only mode. In other words, graphical elements such as graphical elements 304a and / or 304b may not be removed from the primary stream display area 302 when the graphical elements 304a and / or 304b do not include a video component. For example, in some implementations, a graphical element (e.g., graphical element 304a or 304b) is removed from the primary stream display area 302 only if the video component of the graphical element is removed and the teleconference data stream associated with the graphical element is generated by or originates from the client computing device 106(1) (e.g., via an input device 148 associated with the client computing device 106(1)).

[0078] Figure 3C to Figure 3D An example of a display 150(1) is depicted, which is shown connected to Figure 1 1 , according to an exemplary implementation, displays an overlay view of the teleconference session 104 in a user OFF state.

[0079] The overlay view of display 150(1) includes a primary stream display area 302 extending over the screen area of ​​display 150(1). In some configurations, the primary stream display area 302 is configured in a manner that dominates the display. In some configurations, the primary stream display area 302 can be substantially edge-to-edge. As outlined above, the overlay view can optionally include a secondary stream display area based on one or more conditions. In some configurations, the primary stream display area 302 can be larger than the secondary stream display area (see FIG. 4A to FIG. 4D ). Figure 3C to Figure 3D The example of FIG. 300 shows an example of an overlay view without displaying the auxiliary stream display area. A pointer 310 is provided to allow the user to interact with the view. The primary stream display area 302 is divided into four image elements 304a-d, each corresponding to a user media stream. The user media stream may include an audio component, an audio and video component, or an audio component, and an image (e.g., an avatar or picture) representation of the user participating in the teleconference session. Figure 3F An exemplary presentation of an audio component and an image (eg, an avatar or picture) is shown in FIG. See in particular graphical element 304 c.

[0080] Four graphical elements 304a-304d are shown occupying Figure 3C to Figure 3DThe user interface in the illustrated example is arranged with a main stream display area 302; however, any number of graphical elements may be displayed. In some examples, the number of graphical elements displayed may be limited to a maximum value for available bandwidth or as desired to limit the video cluster on display 150(1). In some implementations, when fewer than four participants are involved in a teleconference session, fewer than four graphical elements 304a-304c may be displayed. In a teleconference session involving more than the maximum number of graphical elements, the displayed graphical elements 304a-d may correspond to a dominant person or something that is considered an "active participant." The designation of "active participant" may be predefined as a reference to a particular person, or in some implementations, functionality may be provided to identify "active participants" versus passive participants by applying activity level priorities.

[0081] Activity level priorities rank participants based on their potential contribution to the conference call. In an exemplary implementation, activity level priorities for identifying active versus passive participants may be determined by analyzing each participant's media stream at server 136. The conference call system may include functionality for comparing the activities of the participants and dynamically promoting those who speak more frequently or move and / or speak more frequently as active participants.

[0082] The placement of the graphical elements 304a-304d may also reflect the activity level priority of the participants to which the graphical elements correspond. For example, the overlay view may be defined to have a convention where the top of the main stream display area 302 displays the graphical element 304d corresponding to the most prominent participant. In some sessions, the dominant participant may be the host.

[0083] In some implementations, graphical element 304a is presented in first display area 306. First display area 306 is illustrated by a dashed box. Additionally, graphical element 304b is presented in second display area 308. Second display area 308 is also illustrated by a dashed box. Additionally, graphical element 304c is presented in third display area 312, and graphical element 304d is presented in fourth display area 314. Third display area 312 and fourth display area 314 are also illustrated by dashed boxes, respectively. The dashed boxes do not appear on display 150(1).

[0084] In some implementations, graphical element 304a includes a video component and an audio component. An event may occur that causes the video component associated with graphical element 304a to be removed. The event may be a user-induced event, such as a user-toggled control associated with a user interface of client computing device 106(1), an event caused by a client computing device (e.g., client computing device 106(1)), or an event caused by device 110. In one example, client computing device 106(1) or system 100 may trigger an event that causes the video component associated with graphical element 304a to be removed when it is determined that the communication bandwidth does not provide reliable or user-consumable video.

[0085] System 100 may remove graphical element 304a from primary stream display area 302. Specifically, client computing device 106(1) and / or device 110 may remove graphical element 304a from primary stream display area 302 in response to removing the video component associated with graphical element 304a.

[0086] In some implementations, once the graphical element 304a is removed, the primary stream display area 302 is rearranged to present one or more teleconference data streams in the first display area 306. For example, the graphical element 304b may be enlarged (see Figure 3D ), so that graphical element 304b is at least partially presented in first display area 306. In another example, graphical element 304b remains in second display area 308, and one or more teleconference data streams are presented in first display area 306 at the location of removed graphical element 304a. In a specific example, after graphical element 304a is removed, the one or more teleconference data streams presented in first display area 306 include video and audio components. In another example, after graphical element 304a is removed, the one or more teleconference data streams presented in first display area 306 include an audio component and an avatar or image component.

[0087] In some implementations, the graphical element 304a removed from the first display area 306 is associated with a teleconference data stream generated or originated by a client computing device 106(1) associated with the display 150(1), also referred to herein as an "ME" user interface element or graphical element. In some implementations, a graphical element such as graphical element 304a, 304b, 304c, and / or 304d may not be removed from the primary stream display area 302 when the graphical element 304a, 304b, 304c, and / or 304d does not include a video component. For example, in some implementations, a graphical element (e.g., graphical element 304a, 304b, 304c, or 304d) is removed from the primary stream display area 302 only when the video component of the graphical element is removed and the teleconference data stream associated with the graphical element is generated by or originated from the client computing device 106(1).

[0088] Figure 3E to Figure 3F An example of a display 150(1) is depicted, which is shown connected to Figure 1 The interface 134 of the client computing device 106 (1) in FIG. 1 shows the user closing (OFF) state ( Figure 3E ) and user ON state ( Figure 3F ) overlay view of conference call session 104 under.

[0089] The overlay view of the display 150(1) includes a primary stream display area 302 extending over the screen area of ​​the display 150(1). In some configurations, the primary stream display area 302 is configured in a manner that dominates the display. In some configurations, the primary stream display area 302 can be substantially edge-to-edge. As outlined above, the overlay view can optionally include a secondary stream display area based on one or more conditions (see FIG. 4A to FIG. 4D ). In some configurations, the primary stream display area 302 may be larger than the secondary stream display area. Figure 3E to Figure 3F The example of FIG. 300 shows an example of an overlay view without displaying the auxiliary stream display area. A pointer 310 is provided to allow the user to interact with the view. The primary stream display area 302 is divided into four image elements 304a-d, each corresponding to a user media stream. The user media stream may include an audio component, an audio and video component, or an audio component, and an image (e.g., an avatar or picture) representation of the user participating in the teleconference session.

[0090] Four graphical elements 304a-304d are shown occupying Figure 3E to Figure 3FThe user interface in the illustrated example is arranged with a main stream display area 302; however, any number of graphical elements may be displayed. In some examples, the number of graphical elements displayed may be limited to a maximum value for available bandwidth or as desired to limit the video cluster on display 150(1). In some implementations, when fewer than four participants are involved in a teleconference session, fewer than four graphical elements 304a-304c may be displayed. In a teleconference session involving more than the maximum number of graphical elements, the displayed graphical elements 304a-d may correspond to a dominant person or something that is considered an "active participant." The designation of "active participant" may be predefined as a reference to a particular person, or in some implementations, functionality may be provided to identify "active participants" versus passive participants by applying activity level priorities.

[0091] Activity level priorities rank participants based on their potential contribution to the conference call. In an exemplary implementation, activity level priorities for identifying active versus passive participants may be determined by analyzing each participant's media stream at server 136. The conference call system may include functionality for comparing the activities of the participants and dynamically promoting those who speak more frequently or move and / or speak more frequently as active participants.

[0092] The placement of the graphical elements 304a-304d may also reflect the activity level priority of the participants to which the graphical elements correspond. For example, the overlay view may be defined to have a convention where the top of the main stream display area 302 displays the graphical element 304d corresponding to the most prominent participant. In some sessions, the dominant participant may be the host.

[0093] In some implementations, graphical element 304a is presented in first display area 306. First display area 306 is illustrated by a dashed box. Additionally, graphical element 304b is presented in second display area 308. Second display area 308 is also illustrated by a dashed box. Additionally, graphical element 304c is presented in third display area 312, and graphical element 304d is presented in fourth display area 314. Third display area 312 and fourth display area 314 are also illustrated by dashed boxes, respectively. The dashed boxes do not appear on display 150(1).

[0094] In some implementations, graphical element 304a includes a video component and an audio component. An event may occur that causes the video component associated with graphical element 304a to be removed. The event may be a user-induced event, such as a user-toggled control associated with a user interface of client computing device 106(1), an event caused by a client computing device (e.g., client computing device 106(1)), or an event caused by device 110. In one example, client computing device 106(1) or system 100 may trigger an event that causes the video component associated with graphical element 304a to be removed when it is determined that the communication bandwidth does not provide reliable or user-consumable video.

[0095] System 100 may remove graphical element 304a from primary stream display area 302. Specifically, client computing device 106(1) and / or device 110 may remove graphical element 304a from primary stream display area 302 in response to removing the video component associated with graphical element 304a.

[0096] In some implementations, once graphical element 304a is removed, primary stream display area 302 is rearranged to present one or more teleconference data streams in first display area 306. For example, graphical element 304c may be presented in first display area 306. In one example, graphical element 304e may include an audio component and an avatar or image component. In another example, graphical element 304e may include a video and an audio component. In another example, multiple other graphical elements may be presented in first display area 306.

[0097] In some implementations, the graphical element 304a removed from the first display area 306 is associated with a teleconference data stream generated or initiated by a client computing device 106(1) associated with the display 150(1), also referred to herein as an "ME" user interface element or graphical element.

[0098] In some implementations, graphical elements such as graphical elements 304a, 304b, 304c, and / or 304d may not be removed from primary stream display area 302 when graphical elements 304a, 304b, 304c, and / or 304d do not include a video component. For example, in some implementations, a graphical element (e.g., graphical element 304a, 304b, 304c, or 304d) may be removed from primary stream display area 302 only if the video component of the graphical element is removed and the teleconference data stream associated with the graphical element is generated by or originates from client computing device 106(1). For example, in Figure 3F, graphical element 304c is shown as having been transitioned to a presentation of an audio component and an image (e.g., an avatar or picture). However, according to some implementations, graphical element 304c is not removed from primary stream display area 302 because the teleconference data associated with the presentation of graphical element 304c is not generated or initiated by client computing device 106(1) associated with display 150(1). In other words, when a remote participant transitions to an audio-only state, the associated graphical representation or avatar may not be removed, whereas when a local participant transitions to an audio-only state, the "ME" presentation may be removed.

[0099] In some implementations, when the user moves the FIG. 3A to FIG. 3F The transition to the user ON state may be triggered when the client computing device 106 detects the input device action or any other type of input, and in response to the input, the device 106(1) may responsively send a state change indicator to the server 110 to change the view from ON to OFF. FIG. 3A to FIG. 3E The user close view shown in the is changed to Figure 3F The user opens the view shown in . Figure 3F The user-activated view shown in FIG. 1 may also be displayed as an overlay associated with other views described herein and / or shown in the accompanying drawings.

[0100] The user-activated state may also cause other user interface controls to appear on display 150(1). For example, display 150(1) may include view controls 314. View controls 316 may include control elements (e.g., buttons or switches) that can be used to change the view of the conference call session. For example, the user may wish to switch to a persistent view or a multitasking view. View controls 314 may also include tools for controlling visual aspects of the display or other aspects of the session.

[0101] The user-activated state may also cause a set of core controls 330 to appear, such as Figure 3F . In the example shown, the view on display 150(1) includes a collection of core controls 330, which can be configured to control various aspects of teleconference session 104. For example, a first button of core controls 330 can disconnect device 106(1) from teleconference session 104. A second button of core controls 330 can control the microphone of device 106(1), i.e., a mute button. A third button of core controls 330 can control the camera of device 106(1), i.e., turn the camera on or off. A fourth button of core controls 330 can be used to add users to session 104. In response to receiving actuation of the fourth button by the user, a menu can be displayed that enables the user to select other users as conference participants.

[0102] These examples of core controls 330 are provided for illustrative purposes only and should not be construed as limiting. It should be understood that core controls 330 may relate to any appropriate functionality associated with teleconference session 104. For example, one or more buttons of core controls 330 may change bandwidth limitations, display properties, and the like associated with video data streams. In some configurations, the display of core controls 330 and / or view controls 316 may fade out within a predetermined time period, similar to other controls that fade out after the time period described above. In such a configuration, core controls 330 and / or view controls 316 may be redisplayed based on one or more actions (e.g., hovering or other appropriate user gestures within a predetermined area of ​​the user interface). Core controls 330 and / or view controls 316 may not be displayed in a user-closed state, or may not be displayed after a predetermined time period. In some configurations, core controls 330 and / or view controls 316 may remain visible in the main stream display area 302 of the display, or may remain visible in another area of ​​the display as desired.

[0103] Figure 3G , Figure 3H and Fig. 3I Several user interface processes that may be used when initiating a call are shown.For ease of illustration, consider an example where a first user calls a second user. Figure 3G A first example is shown of a graphical user interface displayed when the first user is "audio only, eg, no video component is shared." Figure 3H A second example is shown of a graphical user interface displayed when a first user shares a video component as well as an audio component.

[0104] In the first example, if Figure 3G As shown in , when the first user is making a call (e.g., during dialing or during the ringing phase), a first user interface arrangement is displayed to the first user. In this example, the user interface includes a graphical representation 390 of the second user. The graphical representation 390 may include an image, a drawing, or any other suitable graphical element. In some configurations, the graphical representation 390 is in the center of the display.

[0105] In the second example, if Figure 3HAs shown in , when the first user is making a call (e.g., during dialing or during the ringing phase), a first user interface arrangement is displayed to the first user. In this example, the user interface includes a graphical representation 390 of the second user. The graphical representation 390 may include an image, a drawing, or any other appropriate graphical element. In some configurations, the graphical representation 390 may be off-center, e.g., lowered to display a background image. In this example, the user interface also includes a live video feed of the first user as a background image. This embodiment enables the first user to see a preview of his or her video feed before connecting to the telephone conference session.

[0106] In both examples, once the second user or group of people has established a connection with the first user, the computing device can display Fig. 3I The user interface shown in . Fig. 3I In the illustrated illustration, the second user is displayed in the primary stream display area 302 and the first user is displayed in the secondary stream display area 315. This configuration is described herein as an embodiment where the overlay view, secondary stream display area 315 times out after a predetermined period of time. For example, the presentation of the first user 304b may be removed from the user interface after a predetermined period of time. The presentation of the first user 304b may be removed from the user interface based on other events (e.g., removal of the video component of the stream). In another embodiment, the presentation of the first user 304b may be removed from the user interface. Figure 3G or any of the interfaces shown in 3H, the computing device may display Figure 3A , wherein the first user is displayed in the first display area 306 and the second user is displayed in the second display area 308. From this state in the teleconference session 104, the computing device can transition to other user interface arrangements as described herein. For example, when the first user transitions to the audio-only mode after the call is established, Figure 3A or Fig. 3I The user interface shown in can be transformed to Figure 3B .

[0107] FIG. 4A to FIG. 4B An example of a persistent view for displaying aspects of a teleconference session 104 is shown. A core control 330 may be used to switch to a persistent view. In addition, a core control 330 may be used to switch to a FIG. 3A to FIG. 3F The overlay view shown in .

[0108] Figure 4A An example of a display 150(1) in a persistent view is depicted in which graphical elements 420a-420d are displayed in a main stream display area 402, also referred to as a "stage." The main stream display area 402 may be displayed in a manner similar to that described in connection with the present invention. FIG. 3A to FIG. 3F The graphical elements 420a-420d are filled in the same manner as described for the main stream display area 302. In addition, similar to FIG. 3A to FIG. 3F The main stream display area 402 is not affected by the graphical elements shown in FIG. FIG. 4A to FIG. 4B The limit on the number of graphical elements that can be represented.

[0109] The primary stream display area 402 has an area that dominates the display 150 ( 1 ), while leaving room for the secondary stream display area 415 to be displayed simultaneously with the primary stream display area 402 . FIG. 4A to FIG. 4B The persistent view in provides a view of the primary stream display area 402 and the secondary stream display area 415 that substantially cover the screen area of ​​the display 150 ( 1 ).

[0110] The auxiliary stream display area 415 may include graphic elements displayed as thumbnails or icons. Figure 4A As shown in , the auxiliary stream display area 415 may include multiple graphical elements 406a-d displayed as thumbnails. These thumbnails can present video and audio components, audio components with images (e.g., avatars or pictures), or audio components. In addition, one of the multiple graphical elements 406a-d displayed as thumbnails may include an "ME" graphical element (e.g., graphical element 406a) displayed as an icon or thumbnail. The remaining multiple graphical elements 406b-d may be active participants in the teleconference session 104. As described above, the graphical elements 406a-d can be displayed as thumbnails that operate as miniaturized video screens, avatars with audio, or simply display boxes with audio.

[0111] The auxiliary stream display area 415 is displayed below the primary stream display area 402 so as not to cover the primary stream display area 402. The auxiliary stream display area 415 is displayed below the primary stream display area 402, but can be located anywhere on the display relative to the primary stream display area 402. For example, the display 150(1) can be rotated to a landscape view. In the landscape view, the auxiliary stream display area 415 can be located, for example, to the right or left of the primary stream display area 402.

[0112] Auxiliary stream display area 415 may include overflow display area 418. Overflow display area 418 may include a quantity value. The quantity value may indicate the number of conference call session participants that are not displayed as graphical elements on display 150(1).

[0113] The graphical elements 420a-420d and 406a-406d can be represented as image or video thumbnails that display images or videos corresponding to media streams sent by users associated with the active participant user interface elements 420a-420d and 406a-406d. The image or video thumbnail can be implemented as a miniature video or image screen having any appropriate aspect ratio such as: for example, 16:9, 4:3, 3:2, 5:4, 5:3, 8:5, 1.85:1, 2.35:1, or any aspect ratio deemed appropriate in a particular implementation. The graphical elements 420a-420d and 406a-406d can be sorted or prioritized as described herein. The positions of the participant user interface elements 420a-420d and graphical elements 406a-406d can indicate ranking or priority based on activity level priority.

[0114] System 100 can determine an activity level priority for each participant in the teleconference session and include a graphical element having a media stream from the participant having the highest activity level priority. The activity level priority can be used to promote or demote participants, thereby providing a dynamic presentation of the participants in display 150(1). For example, in some implementations, as the associated participant becomes more active, the participant representation in auxiliary stream display area 415 can be moved into primary stream display area 402. Similarly, in some implementations, as the associated participant becomes less active, the participant representation in primary stream display area 402 can be moved into auxiliary stream display area 415.

[0115] Figure 4B A view of display 150(1) is depicted with modifications to auxiliary stream display area 415. As shown, overflow display area 418 has been replaced with graphical element 406e. Additionally, graphical element 406a has been changed.

[0116] In some implementations, Figure 4A 1 includes a video component and an audio component. An event may occur that causes the video component associated with graphical element 406a to be removed. The event may be a user-initiated event, such as a user switching a control associated with a user interface of client computing device 106(1), an event caused by a client computing device (e.g., client computing device 106(1)), or an event caused by device 110. In one example, client computing device 106(1) or system 100 may trigger an event that causes the video component associated with graphical element 406a to be removed when it is determined that the communication bandwidth will not provide reliable or user-consumable video.

[0117] The system 100 may remove the graphical element 406a from the auxiliary display area 415 and replace the removed graphical element 406a with Figure 4B Specifically, client computing device 106(1) and / or device 110 may remove graphical element 406a from auxiliary stream display area 415 in response to removal of the video component associated with graphical element 406a.

[0118] In one example, Figure 4B The new graphical element 406a shown in FIG. 4 may include an audio component and an avatar or image component. Figure 3F An exemplary graphical element including an audio component and an avatar or image component is shown in FIG. 304 (see graphical element 304 c ). In another example, new graphical element 406 a may include both video and audio components.

[0119] In some implementations, Figure 4A The graphical element 406a shown removed from the auxiliary stream display area 415 is associated with a conference call data stream generated or initiated by a client computing device 106(1) associated with the display 150(1), and is also referred to herein as an "ME" user interface element or graphical element.

[0120] In some implementations, when a graphical element does not include a video component, the graphical element is not removed from auxiliary stream display area 415. For example, in some implementations, a graphical element may be removed from auxiliary stream display area 415 only if the video component of the graphical element is removed and the teleconference data stream associated with the graphical element is generated by or originated from the client computing device 106(1).

[0121] Figure 4C to Figure 4D Another example of a persistent view including teleconference data is shown. The core control 330 can be used to switch to the persistent view. In addition, the core control 330 can also be used to switch to the FIG. 3A to FIG. 3F The overlay view shown in .

[0122] exist Figure 4C to Figure 4D For the same components with the same function, keep FIG. 4A to FIG. 4B The reference numerals of the elements in . Figure 4C The display 150(1) in FIG. 1 includes a view having a primary stream display area 402. However, in this view, the display 150(1) includes a content media stream 420 presented in the primary stream display area 402 instead of graphical elements.

[0123] In some configurations, a server or client computer can determine when content (e.g., content media stream 420) is part of the teleconference data. For example, a server can determine when a client computing device is sharing content media (e.g., a file, an image of an application, an application shared screen, or any other type of content). Based on determining when content is part of the teleconference data or is not part of the teleconference data, one or more of the views depicted in the figures can be provided.

[0124] Figures 5A to 5H is a screenshot view corresponding to a display of one of the client computing devices in a teleconference session, which shows a user interface arrangement with a drill-down feature. In general, the drill-down feature enables a user to navigate two different user interfaces with a floating monitor view. In some configurations, the floating monitor view can be controlled by one or more events (e.g., an event when a user switches to an audio-only mode).

[0125] In a first example, multiple users are participating in a conference call session 104. Figure 5A As shown in , multiple presentations of users 304a-304d are displayed on the user interface. The user interface may also include one or more optional graphical elements for receiving commands. For example, a first graphical element 383 may be configured to convert the user interface to a chat view in response to a user selection. A second graphical element 384 may be configured to convert the user interface to another view (e.g., a user list) in response to a user selection. Figure 5B An example of a chat view 380 that can be displayed in response to a selection of a first graphical element 383 is shown. In this example, the chat view 380 includes text chat content. The chat view 380 includes a teleconference session monitor view 386 for presenting at least one stream of the teleconference session 104. The selection of a stream can be based on content or a person's activity level. In this example, the presentation of participant 304c is shown. It is also shown that the chat view 380 may include an "ME" view (e.g., a presentation of local participant 304b) for showing live video of local participants. In some configurations, the presentation of the "ME" view can be removed based on one or more events. In one example, if a local user (e.g., a user viewing a stream of a local camera) switches to an audio-only mode, the presentation of the "ME" view can be removed. Figure 5C Another version of the chat view 380 user interface is shown when the "ME" view is removed in response to the local user switching to audio-only mode. The chat view 380 user interface also includes a control graphical element 385 for switching the user interface back to the ME view. Figure 5A The conference call session view shown in .

[0126] exist Figures 5D to 5H In the second example shown in , the device can switch from a teleconference session view to a participant information view. Figure 5D As shown in , multiple presentations of users 304a-304d are displayed on the conference call view user interface. The conference call session view user interface may also include one or more selectable graphical elements for receiving commands. For example, the second graphical element 384 may be configured to convert the conference call view user interface to a participant information view in response to a user selection. Figure 5E An example of a participant information view 387 that may be displayed in response to selection of the second graphical element 384 is shown. In this example, the participant information view 387 includes the names of the participants of the teleconference session 104. The participant information view 387 may also display images associated with the participants. In some configurations, the participant information view 387 may include a search bar 389 for receiving text entries. When the user enters text in the search bar 389, the names of other users who may not be part of the teleconference session 104 may be displayed.

[0127] The participant information view 387 may also include a teleconference session monitor view having a presentation of at least one stream 386 of the teleconference session 104. The selection of a stream may be based on content or the activity level of a person. In this example, the presentation of participant 304c is shown. It is also shown that the participant information view 387 may include an "ME" view (e.g., a presentation of local participant 304b) for showing live video of local participants. In some configurations, the presentation of the "ME" view may be removed based on one or more events. In one example, if a local user (e.g., a user viewing a stream of a local camera) switches to audio-only mode, the presentation of the "ME" view may be removed. Figure 5E Another version of the participant information view 387 user interface is shown when the "ME" view is removed in response to the local user switching to audio-only mode. Figure 5E The user interface also includes a control graphic element 385 for switching the user interface back to Figure 5D The conference call session view shown in .

[0128] Participant information view 387 can also be configured to enable the user to switch to content or people. For example, the user can select the name of the participant by using the pointing device 310 or a touch screen feature. Such an event can cause the computing device to display a third view (e.g., Figure 5G). Contact view 391 can display information related to a selected user. Contact view 391 can also include a teleconference session monitor view having a presentation of at least one stream 386 of teleconference session 104. The selection of a stream can be based on content or activity level of a person. In this example, a presentation of participant 304c is shown. Also shown is that contact view 391 can include an "ME" view (e.g., a presentation of local participant 304b) for showing live video of local participants. In some configurations, if a local user (e.g., a user viewing a stream from a local camera) switches to audio-only mode, the presentation of the "ME" view can be removed. Figure 5H Another version of the contact view 391 user interface is shown when the "ME" view is removed in response to the local user switching to audio-only mode. The contact view 391 also includes a control graphical element 385 for switching the user interface back to the "ME" view. Fig. 5F The conference call session view shown in .

[0129] Now go to Figure 6 , which shows and describes various aspects of a routine 600 for displaying teleconference data on a display of a client computing device 106. It should be understood that the operations of the methods disclosed herein do not necessarily have to be presented in any particular order, and it is possible and expected that some or all of the operations may be performed in an alternative order. For ease of description and illustration, the operations are presented in the order shown. Operations may be added, omitted, and / or performed simultaneously without departing from the scope of protection of the appended claims. In addition, it should be understood that routine 600 may be implemented by one or more of the elements associated with system 100.

[0130] It should also be understood that the illustrated method can be terminated at any time and need not be fully executed. Some or all of the operations of the method and / or substantially equivalent operations, as defined below, can be performed by executing computer-readable instructions included on a computer storage medium. The term "computer-readable instructions" and its variants as used in the specification and claims are used broadly herein to include routines, applications, application modules, program modules, programs, components, data structures, algorithms, and the like. Computer-readable instructions can be implemented on various system configurations including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based programmable consumer electronics, combinations thereof, and the like.

[0131] It should be understood that the logical operations described herein are implemented as: (1) a sequence of computer-implemented actions or program modules running on a computing system; and / or (2) interconnected machine logic circuits or circuit modules within the computing system. The implementation is a matter of choice depending on the performance and other requirements of the computing system. Therefore, the logical operations described herein are variously referred to as states, operations, structural devices, actions, or modules. These operations, structural devices, actions, and modules can be implemented in software, firmware, dedicated digital logic, and any combination thereof.

[0132] For example, the operations of routine 600 are described herein as being performed at least in part by an application, component, and / or circuit (e.g., a system 100 hosting a conference call session 104). Figure 1 The server module 136 may be implemented as a server module 136 in the device 110 in the embodiment of the present invention. In some configurations, the server module 136 may be a dynamic link library (DLL), a static link library, a function generated by an application programming interface (API), a compiler, an interpreter, a script, or any other executable instruction set. Data and / or modules (e.g., the server module 136) may be stored in a data structure in one or more memory components. Data may be retrieved from the data structure by addressing a link or a reference to the data structure.

[0133] Although the following instructions can refer to Figure 1 and Figure 2 Although the components of routine 600 are described herein, it should be understood that the operations of routine 600 may be implemented in many other ways. For example, routine 600 may be implemented, at least in part or in a modified form, by a processor of another remote computer or local circuitry (e.g., client module 130 in client computing device 106(1)). In addition, one or more of the operations of routine 600 may alternatively or additionally be implemented, at least in part, by a chipset working alone or in conjunction with other software modules. In the operations described herein, any service, circuit, or application suitable for providing the techniques disclosed herein may be used.

[0134] See also Figure 6 , routine 600 begins at block 602, where server module 136 receives a plurality of streams 142(1)-142(N) from corresponding client computing devices 106(1)-106(N). A user of each client computing device 106(1)-106(N) transmits a request to join teleconference session 104 and transmits media stream 142 once authorized to participate in teleconference session 104. Server module 136 receives stream 142 from each client computing device 106(1)-106(N).

[0135] At block 604, portions of the media streams are combined to generate teleconference data 146 corresponding to the selected client computing device 106(1) having the display device 150(1). In some configurations, step 604 may involve the following operations: The server and / or the client computer may analyze the teleconference data or the media stream to determine the presence of content. For example, the server may determine when the client computing device is sharing content media (e.g., a file, an image of an application, an application shared screen, or any other type of content). Based on determining when content is part of the teleconference data or is not part of the teleconference data, one or more interface arrangements may be provided.

[0136] At block 606, the teleconference data is configured to be displayed in a user interface arrangement, which may correspond to the user interface arrangement described in reference to FIG. FIG. 3A to FIG. 3F Alternatively, the teleconference data is configured to be displayed in a user interface arrangement, which may correspond to the user interface arrangement described in reference FIG. 4A to FIG. 4D The persistent view described.

[0137] In some configurations, the user interface arrangement can include presentation of multiple streams of teleconference data 146. For example, the user interface arrangement can include a first display area and a second display area. The first and second display areas can be configured to each include presentation of one or more of the multiple streams of teleconference data 146. In one implementation, the first display area is configured to present a first stream of the multiple streams and the second display area is configured to present a second stream of the multiple streams. The user interface arrangement can include a primary stream display area and / or a secondary stream display area.

[0138] At block 608, the teleconference data 146 is sent to the selected client computing device 106(1) for display. Once displayed, the user may view / participate in the teleconference session 104.

[0139] At block 610, an event is detected to remove a video component from a first stream of the plurality of streams. In some implementations, the first stream of the plurality of streams includes teleconference data generated by or originating from the client computing device 106(1).

[0140] At block 612, the user interface arrangement is reconfigured. In one implementation, the system 100 reconfigures the user interface arrangement to remove presentation of the first stream from the first display area. In some implementations, the removed first stream is replaced with at least one of the plurality of streams of teleconference data 146. In some implementations, the at least one of the plurality of streams of teleconference data 146 used to replace the removed first stream includes an audio component, an audio component and an image component (e.g., an avatar or picture), or a video and an audio component.

[0141] At block 614, the reconfigured user arrangement is displayed on the client computing device 106(1).Examples of reconfigured user arrangements are shown in the figures and described herein.

[0142] The disclosure presented herein can be considered in light of the following examples.

[0143] Example 1: A method includes: sending teleconference data including multiple streams to at least one client computing device, the multiple streams including at least a first stream and a second stream, each of the multiple streams including a video component and an audio component; causing the at least one client computing device to display a user interface arrangement, the user interface arrangement including a first display area including a presentation of the first stream and a second display area including a presentation of the second stream; detecting an event to remove the video component from the first stream; in response to detecting the event to remove the video component from the first stream, causing the at least one client computing device to remove the presentation of the first stream from the first display area and present one or more of the multiple streams in at least a portion of the first display area, the one or more streams including a video component and an audio component.

[0144] Example 1: The method of example 1, wherein the first stream is generated by one or more input devices of the at least one client computing device displaying the user interface arrangement including the first display area and the second display area.

[0145] Example 3: A method according to Example 1 and Example 2, wherein the one or more streams include the second stream, and in response to detecting the event of removing the video component from the first stream, causing the at least one client computing device to enlarge the second stream so that the second stream is presented in the at least a portion of the first display area.

[0146] Example 4: The method of Examples 1, 2, and 3, wherein the one or more streams presented in the at least a portion of the first display area is a third stream of the plurality of streams.

[0147] Example 5: A method according to Examples 1, 2, 3 and 4, wherein causing the at least one client computing device to display the user interface arrangement includes: causing the at least one client computing device to display the user interface arrangement including the multiple display areas, wherein the multiple display areas include the first display area, the second display area and an overflow display area, and the overflow display area includes a presentation indicating the number of at least a plurality of the multiple streams.

[0148] Example 6: The method according to Example 5 includes: causing the at least one client computing device to remove the presentation indicating the number of at least the plurality of streams from the overflow display area, and presenting a third stream of the plurality of streams in the overflow display area.

[0149] Example 7: A method according to Examples 1, 2, 3, 4, 5 and 6, wherein the event is at least one of: an event initiated by a user to remove the video component from the first stream; an event to remove the video component from the first stream; or an event initiated by the at least one client computing device to remove the video component from the first stream.

[0150] Example 8: A method includes: receiving, by at least one client computing device, teleconference data comprising multiple streams, the multiple streams comprising at least a first stream and a second stream, each of the multiple streams comprising a video component and an audio component; displaying a user interface arrangement on a display associated with the at least one client computing device, the user interface arrangement comprising a first display area comprising a presentation of the first stream and a second display area comprising a presentation of the second stream; detecting an event to remove the video component from the first stream; in response to detecting the event to remove the video component from the first stream, removing the presentation of the first stream from the first display area, and presenting one or more of the multiple streams in at least a portion of the first display area, the one or more streams comprising a video component and an audio component.

[0151] Example 9: The method of Example 8, wherein the first stream is generated by the at least one client computing device displaying the user interface arrangement including the first display area and the second display area.

[0152] Example 10: A method according to Example 8 and Example 9, wherein the one or more of the multiple streams is the second stream, and in response to detecting the event of removing the video component from the first stream, the second stream is amplified so that the second stream is presented in at least a portion of the first display area.

[0153] Example 11: A method according to Examples 8, 9 and 10, wherein the one or more of the multiple streams presented in at least a portion of the first display area is a third stream of the multiple streams, wherein the third stream is selected based on activity level priority.

[0154] Example 12: A method according to Examples 8, 9, 10 and 11, wherein displaying the user interface arrangement includes: displaying the user interface arrangement including the multiple display areas, wherein the multiple display areas include the first display area, the second display area and an overflow display area, and the overflow display area includes a presentation indicating the number of at least multiple streams among the multiple streams.

[0155] Example 13: The method according to Example 12 includes: removing the presentation indicating the number of at least the plurality of streams from the overflow display area, and presenting a third stream of the plurality of streams in the overflow display area.

[0156] Example 14: A method according to Examples 8, 9, 10, 11, 12 and 13, wherein the event is at least one of: an event initiated by a user to remove the video component from the first stream; an event to remove the video component from the first stream; or an event initiated by the at least one client computing device to remove the video component from the first stream.

[0157] Example 15: A system includes: one or more processing units; and a computer-readable medium having computer-executable instructions encoded thereon, the computer-executable instructions causing the one or more processing units to: receive teleconference data including multiple streams, the multiple streams including at least a first stream and a second stream, each of the multiple streams including a video component and an audio component; display a user interface arrangement, the user interface arrangement including a first display area including a presentation of the first stream and a second display area including a presentation of the second stream; detect an event of removing the video component from the first stream; and in response to the event of removing the video component from the first stream, remove the presentation of the first stream from the first display area, and present one or more of the multiple streams in at least a portion of the first display area, the one or more streams including a video component and an audio component.

[0158] Example 16: The system of Example 15, wherein the first stream is provided by a client computing device displaying the user interface arrangement including the first display area and the second display area.

[0159] Example 17: A system according to Examples 15 and 16, wherein the one or more of the multiple streams is the second stream, and in response to the event of removing the video component from the first stream, the computer-executable instructions cause the one or more processing units to enlarge the second stream so that the second stream is presented in at least a portion of the first display area.

[0160] Example 18: The system of Examples 15, 16, and 17, wherein the one or more of the plurality of streams presented in the at least a portion of the first display area is a third stream of the plurality of streams.

[0161] Example 19: A system according to Examples 15, 16, 17 and 18, wherein the computer-executable instructions cause the one or more processing units to display the user interface arrangement comprising multiple display areas, wherein the multiple display areas include the first display area, the second display area, and an overflow display area, and the overflow display area includes a presentation indicating the number of at least a plurality of the multiple streams.

[0162] Example 20: A system according to Example 19, wherein the computer-executable instructions cause the one or more processing units to remove the presentation indicating the number of at least the plurality of streams from the overflow display area and present a third stream of the plurality of streams in the overflow display area.

[0163] Example 21: A system according to Examples 15, 16, 17, 18, 19 and Example 20, wherein the event is at least one of: a user-initiated event to remove the video component from the second stream; a server-initiated event to remove the video component from the second stream; or a client computing device-initiated event to remove the video component from the second stream.

[0164] Example 22: A method includes: sending teleconference data including multiple streams to at least one client computing device, the multiple streams including a first stream and other streams, each of the multiple streams including a video component and an audio component; causing the at least one client computing device to display a user interface arrangement, the user interface arrangement including a first display area including a presentation of the first stream and a second display area including presentations of one or more of the other streams; detecting an event of removing the video component from the first stream; in response to detecting the event of removing the video component from the first stream, causing the at least one client computing device to remove the presentation of the first stream from the first display area, and reconfigure the user interface arrangement to display the presentation of at least one of the other streams in the first display area.

[0165] Example 23: The method of Example 22, wherein the first stream is generated by the at least one client computing device displaying the user interface arrangement including the first display area and the second display area.

[0166] Example 24: A method according to Example 22 and Example 23, wherein the at least one of the other streams is the one or more of the other streams, and in response to detecting the event of removing the video component from the first stream, the at least one client computing device is caused to zoom in on the one or more of the other streams so that the one or more of the other streams are presented in at least a portion of the first display area.

[0167] Example 25: The method of Examples 22, 23, and 24, wherein the at least one of the other streams presented in the first display area is a second stream of the other streams.

[0168] Example 26: A method includes: sending teleconference data including multiple streams to at least one client computing device, the multiple streams including at least a first stream and a second stream, each of the multiple streams including a video component and an audio component; causing the at least one client computing device to display a user interface arrangement, the user interface arrangement including a first display area including a presentation of the first stream and a second display area including a presentation of the second stream; detecting an event to remove the video component from the first stream, the first stream being generated by an input device of the at least one client computing device; in response to detecting the event to remove the video component from the first stream, causing the at least one client computing device to remove the presentation of the first stream from the first display area and present one or more of the multiple streams in at least a portion of the first display area.

[0169] Example 27: A method according to Example 26, wherein the one or more of the multiple streams is the second stream, and in response to detecting the event of removing the video component from the first stream, causing the at least one client computing device to enlarge the second stream so that the second stream is presented in at least a portion of the first display area.

[0170] Example 28: A method according to Example 26 and Example 27, wherein the user interface arrangement further includes a selectable interface element, wherein the method further includes: in response to receiving a selection of the selectable interface element, causing the at least one client computing device to display a second user interface arrangement, the second user interface arrangement including: a first display area containing descriptions of one or more participants, the presentation of the first stream, and the presentation of a selected stream, wherein the selected stream is based on activity level priority, wherein the second user interface arrangement further includes a second selectable interface element; in response to receiving a selection of the second selectable interface element, causing the at least one client computing device to display the first user interface arrangement.

[0171] Example 29: The method of Example 28, wherein the presentation of the first stream is removed in response to detecting an event to remove a video component from the first stream.

[0172] Example 30: The method according to Example 28 further includes: receiving a selection of a participant from the one or more participants; in response to receiving the selection of the participant, causing the at least one client computing device to display a third user interface arrangement, wherein the third user interface arrangement includes contact information of the participant, wherein the third user interface arrangement also includes a presentation of the first stream and a presentation of the selected stream, and wherein the third user interface arrangement also includes a second selectable interface element for enabling the display of the second user interface arrangement.

[0173] Example 31: The method of Example 30, wherein, in response to detecting the event to remove the video component from the first stream, removing the presentation of the first stream in the third user interface arrangement.

[0174] Example 32: A method according to Examples 26, 27, 28 and 29, wherein the user interface arrangement further includes a selectable interface element, wherein the method further includes: in response to receiving a selection of the selectable interface element, causing the at least one client computing device to display a second user interface arrangement, the second user interface arrangement including: a first display area including a contextual chat associated with one or more participants, the presentation of the first stream and the presentation of a selected stream, wherein the selected stream is based on activity level priority, wherein the second user interface arrangement further includes a second selectable interface element; in response to receiving a selection of the second selectable interface element, causing the at least one client computing device to display the first user interface arrangement.

[0175] Example 33: The method of Example 32, wherein the presentation of the first stream is removed in response to detecting an event to remove a video component from the first stream.

[0176] Although the techniques are described using language specific to structural features and / or methodological acts, it should be understood that the appended claims are not necessarily limited to the described features or acts. Rather, the features and acts are described as example implementations of the techniques.

[0177] The operations of the exemplary processes are shown in each block and summarized with reference to these blocks. These processes are shown as a logical flow of blocks, wherein each block can represent one or more operations that can be implemented with hardware, software, or a combination thereof. In the context of software, these operations represent computer executable instructions stored on one or more computing readable media, which, when executed by one or more processors, enable the one or more processors to perform the described operations. Typically, computer executable instructions include routines, programs, objects, modules, components, data structures, etc. that perform specific functions or implement specific abstract data types. The order in which the operations are described should not be interpreted as limiting, and any number of the described operations can be performed in any order, combined in any order, subdivided into multiple sub-operations, and / or performed in parallel to implement the described process. The described process can be performed by resources associated with one or more devices (e.g., one or more internal or external CPUs or GPUs, and / or one or more hardware logics such as FPGAs, DSPs, or other types of accelerators).

[0178] All methods and processes described above can be embodied by software code modules executed by one or more general-purpose computers or processors, and are fully automated. These code modules can be stored in any type of computer-readable storage medium or other computer storage device. Alternatively, some or all of these methods can be embodied in dedicated computer hardware.

[0179] Unless otherwise specifically stated, conditional language such as "may," "can," "might," or "will," etc., should be understood in context to mean that some examples include certain features, elements, and / or steps, while other examples do not. Therefore, such conditional language is generally not intended to imply that one or more examples require certain features, elements, and / or steps in any way, or that one or more examples must include logic for deciding, whether or not there is user input or prompting, whether or not certain features, elements, and / or steps are included or will be performed in any particular example. Unless otherwise specifically stated, conjunctive language such as the phrase "at least one of X, Y, or Z" should be understood to mean that an item, term, etc. can be X, Y, or Z, or a combination thereof.

[0180] Any routine description, element or block in the flow chart described in this paper and / or the accompanying drawings should be understood to potentially represent module, segmentation or code portion, which include one or more executable instructions for realizing the specific logical function or element in the routine. Alternative implementations are included within the protection scope of the examples described herein, wherein elements or functions can be deleted, or executed in an order different from the order shown or discussed, including substantially synchronous execution or execution in reverse order, depending on the function involved, as understood by those of ordinary skill in the art. It should be emphasized that many changes and modifications can be made to the above examples, and elements therein should be understood to be other acceptable examples. All these modifications and variations are intended to be included within the scope of the present disclosure, and are protected by the appended claims.

Claims

1. A method comprising: receiving, by at least one client computing device, communication data comprising a plurality of streams, the plurality of streams comprising at least a first stream, a second stream, and other remaining streams, each stream in the plurality of streams comprising a video component and an audio component; displaying, on a display device associated with the at least one client computing device, a user interface arrangement comprising: a first display area including a presentation of the first stream and a second display area including a presentation of the second stream; analyzing a video component or an audio component of at least one of the other remaining streams to determine an activity level priority of a participant associated with the at least one stream, wherein the activity level priority is generated for selecting the at least one stream to replace the first stream in response to detecting an event to remove the video component from the first stream, wherein the first stream is transmitted from a first remote device that is separate from a second remote device that transmits the at least one stream, wherein the video component is removed while persistently maintaining a communication connection with the first remote device; detecting the event of removing the video component from the first stream; and In response to detecting the event of removing the video component from the first stream, replacing the presentation of the first stream by removing the presentation of the first stream from the first display area and presenting at least one of the other remaining streams in the first display area, wherein the at least one stream is selected from the other remaining streams based on an activity level priority of a participant associated with the at least one stream.

2. The method of claim 1, wherein: The first stream is generated by the at least one client computing device displaying the user interface arrangement including the first display area and the second display area.

3. The method of claim 1, wherein: The at least one stream is a third stream of the plurality of streams.

4. The method of claim 1, wherein: Displaying the user interface arrangement includes: displaying the user interface arrangement including multiple display areas, wherein the multiple display areas include the first display area, the second display area and an overflow display area, and the overflow display area includes a presentation indicating the number of at least multiple streams among the multiple streams.

5. The method of claim 1, wherein: The event is at least one of: an event initiated by a user to remove the video component from the first stream; An event to remove the video component from the first stream; or an event initiated by the at least one client computing device to remove the video component from the first stream.

6. The method of claim 1, wherein: The at least one stream is selected based on a frequency of human speech present in the at least one stream.

7. The method of claim 1, wherein: The at least one stream is selected based on movement of a person represented in the at least one stream.

8. The method of claim 1, wherein: The event of removing the video component from the first stream comprises a determination that the first stream no longer comprises video data.

9. The method of claim 1, wherein: The event of removing the video component from the first stream comprises determining that the first stream has been converted to an audio-only stream.

10. The method of claim 1, wherein: The event that removes the video component from the first stream includes determining that the first stream has switched to an audio-only stream, wherein, in response to determining that the first stream has switched to an audio-only stream, the server initiates presentation that removes the first stream from the first display area.

11. The method of claim 1, wherein: The event of removing the video component from the first stream comprises a remote device terminating transmission of the first stream.

12. A system comprising: one or more processing units; as well as A computer-readable medium having computer-executable instructions encoded thereon, the instructions causing the one or more processing units to perform the following operations: Receiving communication data including a plurality of streams, the plurality of streams including at least a first stream, a second stream, and other remaining streams, each of the plurality of streams including a video component and an audio component; displaying a user interface arrangement, the user interface arrangement comprising: a first display area including a presentation of the first stream and a second display area including a presentation of the second stream; analyzing a video component or an audio component of at least one of the other remaining streams to determine an activity level priority of a participant associated with the at least one stream, wherein the activity level priority is generated for selecting the at least one stream to replace the first stream in response to detecting an event to remove the video component from the first stream, wherein the first stream is transmitted from a first remote device that is separate from a second remote device that transmits the at least one stream, wherein the video component is removed while persistently maintaining a communication connection with the first remote device; detecting the event of removing the video component from the first stream; and In response to detecting the event of removing the video component from the first stream, replacing the presentation of the first stream by removing the presentation of the first stream from the first display area and presenting at least one of the other remaining streams in the first display area, wherein the at least one stream is selected from the other remaining streams based on an activity level priority of a participant associated with the at least one stream.

13. The system of claim 12, wherein: One or more of the plurality of streams presented in at least a portion of the first display area is a third stream of the plurality of streams.

14. The system of claim 12, wherein: The computer-executable instructions cause the one or more processing units to display the user interface arrangement comprising multiple display areas, wherein the multiple display areas include the first display area, the second display area, and an overflow display area, and the overflow display area includes a presentation indicating a number of at least a plurality of the multiple streams.

15. A system as described in claim 14, wherein the computer-executable instructions cause the one or more processing units to remove the presentation indicating the number of at least the plurality of streams from the overflow display area and present a third stream of the plurality of streams in the overflow display area.

16. A method comprising: sending communication data including a plurality of streams to at least one client computing device, the plurality of streams including at least a first stream, a second stream, and other remaining streams, each stream in the plurality of streams including at least a video component and an audio component; causing the at least one client computing device to display a user interface arrangement comprising: a first display area including a presentation of the first stream and a second display area including a presentation of the second stream; analyzing a video component or an audio component of at least one of the other remaining streams to determine an activity level priority of a participant associated with the at least one stream, wherein the activity level priority is generated for selecting the at least one stream to replace the first stream in response to detecting an event to remove the video component from the first stream, wherein the first stream is transmitted from a first remote device that is separate from a second remote device that transmits the at least one stream, wherein the video component is removed while persistently maintaining a communication connection with the first remote device; detecting the event of removing the video component from the first stream, the first stream being generated by an input device of the at least one client computing device; and In response to detecting the event of removing the video component from the first stream, causing the at least one client computing device to replace the presentation of the first stream by removing the presentation of the first stream from the first display area and presenting at least one of the other remaining streams in the first display area, wherein the at least one stream is selected from the other remaining streams based on an activity level priority of a participant associated with the at least one stream.

17. The method of claim 16, wherein: The user interface arrangement further includes selectable interface elements, wherein the method further includes: in response to receiving a selection of the selectable interface element, causing the at least one client computing device to display a second user interface arrangement comprising a first display area including descriptions of one or more participants, a presentation of the first stream, and a presentation of a selected stream, the selected stream being based on an activity level priority, wherein the second user interface arrangement further comprises a second selectable interface element; and In response to receiving a selection of the second selectable interface element, causing the at least one client computing device to display the user interface arrangement.

18. The method of claim 17, wherein: Responsive to detecting an event that removes a video component from the first stream, presentation of the first stream is removed.

19. The method of claim 17, further comprising: receiving a selection of a participant from among the one or more participants; as well as In response to receiving a selection of the participant, causing the at least one client computing device to display a third user interface arrangement, wherein the third user interface arrangement includes contact information of the participant, wherein the third user interface arrangement also includes presentation of the first stream, and presentation of the selected stream, wherein the third user interface arrangement further includes a second selectable interface element for causing display of the second user interface arrangement.

20. The method of claim 19, wherein: Responsive to detecting the event to remove the video component from the first stream, removing presentation of the first stream by the third user interface arrangement.

Images