Aspect Ratio for Dynamic Control of a Communication Session Video Stream

Abstract

The disclosed technology improves user engagement and promotes efficient use of computing resources by providing dynamic control of the aspect ratio for communication session rendering based on the physical orientation of the device. In some configurations, when the device is in a first orientation, such as portrait orientation, the system can select a first aspect ratio for the respective video streams of a communication session. Additionally, when the device is in a second orientation (e.g., landscape orientation), the system can select a second aspect ratio for the respective video streams. In some configurations, the first aspect ratio can be greater than the second aspect ratio, or the aspect ratios can be selected based on a target aspect ratio that can be adjusted over time. By dynamically selecting the aspect ratio for rendering of each stream, the screen space of the device can be optimized while keeping the device in various physical orientations.

Description

Background Art

[0001] There are many different communication systems that allow users to collaborate. For example, some systems allow people to collaborate by using live video streams, live audio streams, and other forms of text-based or image-based media. Participants in a communication session can share a video stream that shows a single person or a group of people and the shared content. Such systems can provide an experience that simulates a face-to-face meeting for the participants in the communication session.

[0002] Although there are many different types of systems that allow users to collaborate, some existing systems have many drawbacks. For example, when an online meeting includes multiple video streams, most existing systems display each stream in a fixed arrangement. Such designs typically include a grid pattern where each rendering has a fixed size and shape. Such a design can also result in scenarios where certain video streams are not displayed to viewers in an optimal way. For example, when a mobile device is held in one orientation, such as a vertical portrait orientation, each video stream can be rendered using a fixed aspect ratio. When the device rotates, such as from a portrait orientation to a landscape orientation, each video rendering can be rearranged and resized, but maintains the same fixed aspect ratio. Such a design may not optimally utilize the screen space for each orientation. A collection of stream renderings with a fixed aspect ratio may be suitable for one set of screen sizes, such as when in a portrait orientation, but not for another set of screen sizes, such as when in a landscape orientation.

[0003] In addition, when a device rearranges the renderings of multiple video streams, if the movement of each rendering is not done in an orderly manner, the movement of multiple video streams may distract the user. Further, when the renderings are resized, such resizing may partially cut some users out of the rendering, especially when the user is in a multi-person video stream. Such design issues may not optimally promote user engagement because viewers may not be able to clearly see everyone, track the progress of a stream, or see important gestures made by everyone.

[0004] Software applications that do not promote user engagement can result in production losses and inefficiencies with respect to computing resources. For example, participants in a communication session (such as an online meeting) may need to refer to recordings or other resources when they miss or overlook content. When a user misses a key point during a live meeting, the content may need to be resent. Such activities can result in inefficiencies in the use of network, processor, memory, or other computing resources. Further, when the level of participant engagement during a meeting is negatively affected, such production losses may result in the need for extended meetings or follow-up meetings, which in turn can consume additional computing resources. Such production losses and inefficiencies related to computing resources are more severe when the system is used to provide a collaborative environment for a large number of participants.

[0005] In addition to losses in user engagement, when a communication system fails to effectively display a person's live video or shared content, it can also lead to many other inefficiencies. Participants may miss important social cues, for example, when a person raises a hand, starts speaking, looks in a certain direction, and so on. These drawbacks sometimes require users to manually interact with others using a separate communication system. For example, if a cue is missed or there is suspicion of a certain type of miscommunication, some users still send text messages or emails to other participants while on a conference call. Such manual steps can disrupt a person's workflow and be very inefficient when helping a person establish a collaboration protocol with a group of people. These drawbacks of existing systems can result in productivity losses and inefficient and redundant use of computing resources. Summary of the Invention

[0006] The techniques disclosed herein improve user engagement and more effectively use computing resources by providing dynamic control of the aspect ratio for communication session rendering based on the physical orientation of the device. In some configurations, when the device is in a first orientation (e.g., portrait orientation), the system can select a first aspect ratio for the respective video streams of the communication session. Additionally, when the device is in a second orientation (e.g., landscape orientation), the system can select a second aspect ratio for the respective video streams. In some configurations, the first aspect ratio can be greater than the second aspect ratio. In other configurations, the first aspect ratio can be greater than a target aspect ratio, and the second aspect ratio can be less than the target aspect ratio. For example, when the device is in a vertical orientation, e.g., when the screen is in portrait orientation, a larger aspect ratio than the target aspect ratio of one (1:1) can be used to display the respective streams. Thus, rendering can be displayed in aspect ratios such as 4:3, 16:9, etc. When the device is in the second orientation, e.g., when the screen is in landscape orientation, a smaller aspect ratio than the target aspect ratio of one (1:1) can be used to display the respective streams, e.g., rendering can be displayed in aspect ratios such as 3:4, 5:9, etc. In some embodiments, the target aspect ratio can be selected based on one or more factors, including the aspect ratio of the display screen, the number of participants depicted in the video stream, etc. In other embodiments, the target aspect ratio can include, for example, a range from 3:4 to 4:3. Thus, in some embodiments, the device can select a first aspect ratio greater than the range when the device is in portrait orientation, or a second aspect ratio less than the range when the device is in landscape orientation. By dynamically selecting the aspect ratio for each stream rendering, in some embodiments, the aspect ratio can be based on the target aspect ratio, and screen space can be optimized when the device is held in various physical orientations.

[0007] In some configurations, the device can select a fixed aspect ratio for a first set of streams depicting a threshold number of people, and the aspect ratio can be adjusted based on the device's orientation for a second set of streams depicting fewer than the threshold number of people. In one example, the system can select a fixed wide aspect ratio for a video stream depicting the threshold number of people and maintain that wide aspect ratio when the device transitions to a different physical orientation. Although the selected rendering has a fixed aspect ratio, the rendering of a single-person (e.g., fewer than the threshold number of people) video stream can be adjusted based on the device's orientation. By affixing the aspect ratio for the stream rendering depicting the threshold number of people while adjusting the aspect ratios of other streams, screen space can be further optimized while keeping the device in various physical orientations.

[0008] In some configurations, the system can analyze the streams of a communication session and select the streams with a threshold number of people. The system can further analyze the selected streams to identify individuals with a threshold level of activity. The system can then scale the image to magnify the individuals with a threshold level of activity. This is an improvement over existing systems that typically involve rigid user interface layouts for mobile devices. One benefit of the currently disclosed technology is to make the active people depicted in a multi-person video stream appear the same size as the people depicted in a single-person video stream. Adjusting the size of the selected stream while adjusting the aspect ratios of other streams helps the system provide more control over the display layout to balance the representation of each person shown within the user interface.

[0009] The features disclosed herein help promote user engagement of presenters and viewers by making the actions of each person in a multi-person video more visually accessible and, in some embodiments, balancing the display of people in a multi-person video with the display of people in a single-person video stream. These techniques can also be applied to any identified object within the video stream, as these techniques are not limited to identifying the number of people depicted in the video stream.

[0010] The examples described herein are provided in the context of a collaborative environment, such as a private chat session, a multi-user content editing session, a group meeting, a live broadcast, etc. For illustrative purposes, it can be appreciated that computer management of a collaborative environment involves any type of computer that manages a communication session where two or more computers share video data, including recorded video streams and live video streams. Additionally, it can be appreciated that the techniques disclosed herein can be applied to any user interface layout for displaying content. The scope of the present disclosure is not limited to embodiments associated with a collaborative environment.

[0011] The techniques disclosed herein provide numerous improvements to existing computers. For example, computing resources such as processor cycles, memory, network bandwidth, and power are used more efficiently because the system can dynamically control the size, position, and shape of video streams. By providing user interfaces with dynamically controlled aspect ratios for individual streams based on the physical orientation of the device, the user interfaces can provide more visual detail of objects of interest. Thus, the techniques disclosed herein can provide more efficient use of computing resources by providing user interfaces that optimize user engagement.

[0012] The techniques disclosed herein provide numerous improvements to existing computers. For example, computing resources such as processor cycles, memory, network bandwidth, and power are used more efficiently because the system can dynamically control the size, position, and shape of video streams depicting a threshold number of people. By providing user interfaces with dynamic control that provide more visual detail of objects of interest, the techniques disclosed herein can provide more efficient use of computing resources. The system can improve user interaction with computing devices by reducing the need for additional communication systems because the disclosed system can reduce or eliminate the need for requests for retransmitted content, duplicate content, etc. Improving user interaction with the device can also result in a reduction in unnecessary or redundant input, thereby reducing unintentional input, corrective input, and other types of user interaction that utilize computing resources. Other technical benefits not specifically mentioned herein can also be achieved by practicing the disclosed subject matter.

[0013] Those skilled in the art will also understand that aspects of the subject matter described herein can be practiced on or in conjunction with other computer system configurations other than those specifically described herein, including multiprocessor systems, microprocessor-based or programmable consumer electronics, augmented or virtual reality devices, video game devices, handheld computers, smartphones, smart TVs, autonomous vehicles, smart watches, e-readers, tablet computing devices, dedicated hardware devices, networked devices, and the like.

[0014] Features and technical advantages other than those explicitly described above will become apparent by reading the following detailed description and viewing the associated drawings. This summary is provided to introduce a selection of concepts in a simplified form that will be further described below in the detailed description. The summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. For example, the term "technique" can refer to systems, methods, computer-readable instructions, modules, algorithms, hardware logic, and / or operations permitted by the context described above and throughout the document. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The detailed description is described with reference to the accompanying drawings. In the figures, the leftmost digit of a reference number identifies the figure in which the reference number first appears. The same reference numbers in different figures represent similar or identical items. A reference to a single item among multiple items may use a reference number with a letter from an alphabetical sequence to refer to each individual item. A general reference to an item may use a specific reference number without an alphabetical sequence.

[0016] Figure 1 Illustrates aspects of a device for configuring a user interface layout based on the physical orientation of the device.

[0017] Figure 2 Illustrates aspects of a device for configuring a user interface layout based on the physical orientation of the device and the activity of at least one user depicted in the rendering of a video stream.

[0018] Figure 3 Illustrates aspects of a device for configuring a user interface layout based on the physical orientation of the device and the number of people depicted in the rendering of a video stream.

[0019] Figure 4 Illustrates aspects of a device for configuring a user interface layout based on the physical orientation of the device and the number of people depicted in multiple renderings of multiple video streams.

[0020] Figure 5A Illustrates a first state of two user interface layouts based on the number of people depicted in two or more video streams.

[0021] Figure 5B Illustrates a change in the number of people depicted in a video stream received by the device.

[0022] Figure 5C Illustrates a second state of two user interface layouts based on detecting a change in the number of people depicted in a video stream received by the device.

[0023] Figure 6 Is a flowchart illustrating aspects of a routine for efficiently generating a user interface layout.

[0024] Figure 7 Is a computing system diagram showing aspects of an illustrative operating environment for the techniques disclosed herein.

[0025] Figure 8 Is a computing architecture diagram showing aspects of the configuration and operation of a computing device that can implement aspects of the techniques disclosed herein. Detailed Description

[0026] The following detailed description is directed to techniques for improving user engagement and more efficiently using computing resources by providing dynamic control of the aspect ratio for rendering video streams of a communication session based on the physical orientation of a display screen of a device. In some configurations, the system can control the dimensions, sizes, and positions of multiple video renderings based on multiple factors, including but not limited to the physical orientation of the display screen, the number of individuals depicted in the video stream, and / or the activity level of people depicted in one or more video streams. Additional details of the system for determining the order of arranging streams are described herein.

[0027] Figure 1 Aspects of device 100 for configuring a user interface layout are illustrated, the user interface layout having a rendering with a specific aspect ratio based on the physical orientation of the display screen of device 100. Device 100 can receive streams from a server. The server can manage multiple data streams having video components and audio components, thereby allowing device 100 to communicate with multiple other remote devices. The following references Figure 7 System 602 shown provides additional details of the server that manages communication sessions between device 100 and other remote devices. The following also references additional aspects of device 100 in more detail with respect to device 606 shown in Figure 7

[0028] In some configurations, device 100 (also referred to herein as "computing system 100") can receive multiple streams, each stream including a video component. Device 100 can also receive orientation data from sensors mounted to device 100. Additional details regarding the sensors are provided below with respect to Figure 7 The orientation data can indicate whether the display screen 101 of computing system 100 is in a portrait orientation or a landscape orientation. In response to determining that display screen 101 is in a portrait orientation, device 100 can cause a first user interface layout 102A to be displayed, the first user interface layout 102A including respective renderings 103 (individually referred to as 103A - 103D) of respective streams received by device 100. In a portrait orientation, the device can select a first aspect ratio that is greater than a target aspect ratio. For example, the target aspect ratio can be 1:1 (a value of 1). In such an example, the first aspect ratio can be any aspect ratio greater than the value 1, such as a landscape dimension, such as 16:9, 4:3, etc. As shown in the example of Figure 1 with a given target aspect ratio of one (1), and since the device is in a portrait orientation, the aspect ratio of each rendering 103A - 103D has a landscape dimension, such as 16:9.

[0029] As shown in Figure 1 ​As shown, during operation, device 100 may also receive additional orientation information from a sensor. Device 100 may monitor the orientation of the device or the orientation of the display 101 of the device, and in response to determining that the display screen 101 is converted from a portrait orientation to a landscape orientation, device 100 may invoke a conversion of the display from the first user interface layout 102A to a second user interface layout 102B including an updated rendering 103' (individually referred to as 103A'-103D') of each stream, where each of the updated renderings 103 has a second aspect ratio that is less than the target aspect ratio. In some configurations, the second aspect ratio may be less than the first aspect ratio. Thus, in some configurations, the target aspect ratio or target aspect ratio range may not be used.

[0030] The selection of the target aspect ratio may be based on one or more factors. For example, the target aspect ratio or the selected target ratio may be based on the dimensions of the allocated display area. The allocated display area may be a display area specified within one or more display screens. For example, the allocated display area may be a specified display area that spans multiple display devices or a specified display area that is part of a single display device. In one illustrative example, the selected aspect ratio may be selected based on one or more dimensions of a display device in communication with a computing system. Thus, if the display screen has a 16:9 display area, the target aspect ratio or the selected aspect ratio may include a value that is divisible by at least one dimension of the display screen. In this example, the aspect ratio of the device in the portrait orientation may be 9:4, and the aspect ratio of the device in the landscape orientation may be 4:9.

[0031] In another illustrative example, the target aspect ratio may be selected based on a preferred layout for the dimensions of a particular display device. For example, if a user prefers to have a landscape rendering when their device is in the portrait orientation and a portrait rendering when their device is in the landscape orientation, the user may set the target aspect ratio to one (1:1). However, if they wish to bias each aspect ratio towards a wider configuration for both the portrait and landscape orientations, the target aspect ratio may be adjusted to a higher value, such as 4:3. Similarly, if they wish to bias each aspect ratio towards a narrower configuration for both the portrait and landscape orientations, the target aspect ratio may be adjusted to a lower value, such as 3:4.

[0032] In some cases, the target aspect ratio may include a range of aspect ratios. For example, the target aspect ratio may include a range from 3:4 to 4:3. Thus, the target aspect ratio can be used to select a first aspect ratio above the range and a second aspect ratio below the range. Such embodiments enable the device to have more fine-grained control over each user interface layout used in each orientation.

[0033] Now refer to Figure 2, aspects of embodiments for configuring a user interface layout based on a device's physical orientation and the activity level of at least one stream are shown and described below. In some configurations, respective aspect ratios may be applied to select a stream based on the presence of one or more criteria satisfied by the activity of the video component or the audio component of the stream. Figure 2 Aspects of embodiments for configuring a user interface layout based on a device's physical orientation and the activity of at least one stream received by the device 100 are illustrated. In this example, once the device determines that the display screen 10 is in a landscape orientation, a scaling factor is determined for one or more selected streams depicting a threshold number of people. For illustrative purposes, a first rendering 103A generated from a first stream has a video component depicting four people. If the device 100 receives configuration data defining a threshold of three (3) people, the device will select the rendering of the first stream for a modified scaling factor.

[0034] The scaling factor may be selected based on the size of a person having a threshold activity level. For example, if a person within the video is speaking at a threshold rate, speaking at a threshold volume, or performing a gesture that satisfies one or more criteria, the scaling factor may be selected to enable the device to focus on the rendering 103A' of the person's stream, as Figure 2 shown. The system may continue to monitor the activity of the video component or the audio component of the stream of people having a threshold number. Thus, if a first person has a threshold activity level and then stops or reduces that activity, and then a second person begins a threshold activity level, the device may focus the rendering of the stream on the second person and thus change the scaling factor to accommodate the display of the second person.

[0035] In some configurations, a unique scaling factor may be applied to each rendering of each stream. The scaling factor may be selected for each stream to equalize at least one dimension of the physical characteristics of two or more people depicted in different streams. One dimension may include the width and / or height of a person's face, head, top of the head, or any other dimension that can be measured by analyzing the video image of a person. For example, consider a scenario where a first rendering depicts two people and a second rendering depicts one person. In this example, without applying a scaling factor, the people in the two-person video in the rendering appear smaller, e.g., only half the size of the person in the single-person video. To make the display size of each person equal, a scaling factor may be selected to increase the size of the rendering of the two-person video, which may include cropping the edges from the image. Additionally or alternatively, another scaling factor may be selected to reduce the size of the rendering of the single-person video.

[0036] In some configurations, the scaling factor for each video can be selected based on the number of people depicted in each video. The selection of the scaling factor can be linear or non - linear. For example, a two - person video can be magnified by a factor of two, a three - person video can be magnified by a factor of three, and so on. Alternatively, the size of a two - person video can be magnified by a factor of 1.75, the size of a three - person video can be magnified by a factor of 2.15, and so on. In another embodiment, the scaling factor for each video can be selected based on the dimension of at least one physical characteristic of the people depicted. For example, if one person in a two - person video has a measurement from the top of their head to their chin of 2 measurement units, while the person in a single - person video has a measurement from the top of their head to their chin of 4 measurement units, the two - person video can be magnified by two factors. Other scaling factors can be selected based on the measurements according to the desired result. By equalizing or at least partially equalizing at least one dimension of the individual users depicted in each stream, by allowing viewers to view the details of the gestures shown and reducing any visual preference that may be caused by people being displayed in different sizes, the system can help improve user engagement.

[0037] In an illustrative example, the device can analyze multiple incoming streams to determine that at least one of the multiple streams depicts a threshold number of people. The device can then analyze the identified stream that depicts the threshold number of people to identify at least one person associated with a threshold activity level. The device can then determine a scaling factor suitable for generating a modified rendering of the at least one stream, where the scaling factor of the modified rendering is configured to shift the focus to the at least one person associated with the threshold activity level. Shifting the focus to a person can include panning and zooming into a portion of the video rendering that depicts that person.

[0038] Now referring to Figure 3 , other embodiments of the device are shown and described. In some configurations, the aspect ratio of one or more specific renderings can be fixed, while the aspect ratios of other renderings are configured to change with the orientation of the device. In an illustrative example, for a rendering that depicts a threshold number of people, the aspect ratio can be fixed. The aspect ratios of renderings that do not depict the threshold number of people may change according to the orientation of the device. Aspects of such an embodiment are shown in Figure 3 . As shown, the first rendering 103A depicts four people. If the configuration data indicates a threshold of, for example, three people, and the device is rotated, the aspect ratio associated with the content of the first rendering 103A can be fixed. Thus, as shown, when the display screen 101 is in a portrait orientation, an aspect ratio greater than the target aspect ratio is used to display all of the renderings 103A - 103B, for example, using a landscape view to display the renderings 103A - 103B.

[0039] Then, when the device 100 is rotated such that the display screen 101 is in a landscape orientation, the updated rendering 103A' is displayed using the same aspect ratio as the first rendering 103A, since the aspect ratio is fixed for this content. It is also shown that when the device 100 is rotated, the aspect ratios of the other renderings (103B - 103D) are reduced to an aspect ratio less than the original aspect ratio, or below the target aspect ratio.

[0040] In an illustrative example, the device can analyze multiple incoming streams to determine that at least one of the multiple streams depicts a threshold number of people. In response to determining that at least one of the multiple streams depicts a threshold number of people, e.g., 3 or 4 people, the device selects a fixed aspect ratio for the at least one stream. The fixed aspect ratio can be configured to override any association between the identified stream and a second aspect ratio. Thus, when the display screen is in a portrait orientation or a landscape orientation, the rendering of the at least one stream is displayed using the fixed aspect ratio.

[0041] Now referring Figure 4 , embodiments for sorting the renderings of individual streams based on the number of people depicted in the streams are shown and described below. In this embodiment, if the first stream depicts four people, the second stream depicts two people, and the third stream depicts one person, the device will configure the user interface layout to sort the streams from top to bottom as: the first stream, the second stream, and the third stream. Such an arrangement helps the viewer focus on the streams with a particular activity concentration.

[0042] This example is provided for illustrative purposes and should not be construed as limiting. Although the examples described herein sort the renderings from top to bottom based on the number of people, it can be appreciated that the order can be arranged from bottom to top, from left to right, from right to left, or any other ordered configuration. It can also be appreciated that the order of arranging the renderings can be based on other factors, such as the activity level, e.g., the volume or rate at which people are speaking or the manifestation of a particular gesture. In such an embodiment, an activity level can be generated for each stream, and the renderings of each stream can be sorted within the user interface layout 102 based on the activity level.

[0043] The techniques disclosed herein can utilize any suitable techniques to analyze multiple communication streams to determine the number of people depicted in one or more video components. For example, face recognition, pattern recognition, or motion recognition techniques can be used to determine the number of people depicted in the video components of each stream. Alternatively, a remote computer can analyze one or more streams to determine the number of people depicted in the video components of each stream and provide the number of people depicted in each stream by transmitting the results to the device. The device can then configure the first user interface layout or the second user interface layout to sort the respective renderings of each stream based on the number of people depicted in the video components of each stream. The participants can be arranged in descending order of the number of people from left to right, right to left, top to bottom, or bottom to top in the user interface. The order can also follow any other linear path defined in the user interface.

[0044] Figures 5A - 5C Another example of a user interface that can be arranged based on the physical orientation of the device and the number of people depicted in each stream is illustrated. In some embodiments, the device can continuously monitor the number of people depicted in each stream and change the order of each stream as the number of people in each stream changes. Figure 5A It is shown that the rendering 103A of the first stream includes three people and the rendering 103B of the second stream includes two people. Due to the number of people depicted in each stream, when the device is in the portrait orientation, the rendering of the first stream is displayed in the top position and the rendering of the second stream is displayed in the second position from the top position. When the device is in the landscape orientation, the rendering of the first stream is displayed in the left position and the rendering of the second stream is displayed in the second position from the left position.

[0045] As the content of the second stream changes, for example, the second stream has a video component that shows an increasing number of people, the device can detect such a change and modify the order in which the renders are displayed. As Figure 5B shown, the content of the second stream has changed such that the second rendering shows four people. Since the second stream now depicts more people than the first stream, the order of the rendered streams has changed, as Figure 5C shown. As shown, in response to the detected change, the second stream depicts more than the first stream. When the device is held in the portrait orientation, the rendering 103A of the first stream moves from the top to the second position, and the rendering 103B of the second stream moves to the top position. Additionally, in response to the detected change, when the device is held in the landscape orientation, the rendering 103A of the first stream moves from the left to the second position, and the rendering 103B of the second stream is moved to the left position.

[0046] In an illustrative example, a device can analyze multiple streams to determine the number of people depicted in the video component of each stream. The device can configure a first user interface layout or a second user interface layout to position the rendering of the first stream at one end of a rendering sequence of the multiple streams and to position the rendering of the second stream at a second position in the rendering sequence. The sequence can be a row of renderings, an ordered list arranged in rows and columns, etc. In this example, the number of people depicted in the first stream is greater than the number of people depicted in the second stream, as Figure 5A shown

[0047] The device can continue to monitor the streams and determine whether the number of people in each stream changes. If the number of people depicted in the first stream changes or the number of people depicted in the second stream changes, where the number of people depicted in the first stream becomes less than the number of people depicted in the second stream, the device can reconfigure the order of the positioning of the renderings in each user interface layout. Thus, in response to determining when the number of people depicted in the first stream becomes less than the number of people depicted in the second stream, the device can configure the first user interface layout or the second user interface layout to move the rendering of the second stream to one end of the rendering sequence and to position the rendering of the first stream at the second position in the rendering sequence.

[0048] Figure 6 FIG. is a diagram illustrating aspects of routine 500 that improves user engagement and more efficiently uses computing resources by providing dynamic control of the aspect ratio for video stream rendering. Those of ordinary skill in the art will understand that the operations of the methods disclosed herein need not be presented in any particular order, and that performing some or all of the operations in alternative orders is possible and is contemplated. For ease of description and illustration, the operations are presented in the order of demonstration. Operations may be added, omitted, performed together, and / or performed concurrently without departing from the scope of the appended claims.

[0049] It should also be understood that the illustrated methods can end at any time and need not be executed in their entirety. Some or all of the operations of these methods can be performed by executing computer-readable instructions included on a computer storage medium, and / or substantially equivalent operations, as defined herein. As used in the description and claims, the term "computer-readable instructions" and its variants are used herein broadly to include routines, applications, application modules, program modules, programs, components, data structures, algorithms, etc. Computer-readable instructions can be implemented on a variety of system configurations, including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based programmable consumer electronics, combinations thereof, etc.

[0050] Accordingly, it should be appreciated that the logical operations described herein are implemented as (1) a sequence of computer implemented acts or program modules running on a computing system such as those described herein) and / or (2) as interconnected machine logic circuits or circuit modules within the computing system. The implementation is a choice of design depending on the performance and other requirements of the computing system. Accordingly, the logical operations may be implemented in software, firmware, special purpose digital logic, and any combination thereof.

[0051] In addition, Figure 6 the operations shown in this and other figures may be implemented in association with the example presented UI. For example, the various devices and / or modules described herein may generate, send, receive, and / or display data associated with the content of a communication session (e.g., live content, broadcast events, recorded content, etc.) and / or include a rendered presentation UI of one or more participants of a remote computing device, avatar, channel, chat session, video stream, image, virtual object, and / or application associated with the communication session.

[0052] Routine 500 begins at operation 502, where one or more computing modules receive a plurality of streams. As described in more detail below, with respect to Figure 7 , the system may manage a plurality of streams received from a plurality of different client devices. The streams may be bundled and delivered to respective computing devices, which may be used to display different arrangements of each stream. Each stream may include an audio component and a video component.

[0053] Next, at operation 504, one or more computing modules may receive orientation data from sensors installed in the computing device. The orientation data may indicate that the display screen 101 of the computing device 100 is in a portrait orientation or a landscape orientation. As described in more detail below, with respect to Figure 7 , the sensors may include an accelerometer, a compass, or any other device for detecting the physical orientation of the device, e.g., whether the device is in a vertical position, e.g., portrait orientation, or a lateral position, e.g., landscape orientation.

[0054] Next, at operation 506, one or more computing modules may cause a first user interface layout 102A to be displayed, the first user interface layout 102A including respective renderings 103 of respective ones of a plurality of streams. In response to determining that the display screen 101 is in a portrait orientation, each of the respective renderings may each have a first aspect ratio greater than a target aspect ratio. For example, each of the respective renderings may each have a first aspect ratio of 16:9 forming a horizontal dimension, which is greater than a target aspect ratio such as 1:1. It can be appreciated that other target aspect ratios may be selected to provide different results for each layout. Additionally, the target aspect ratio may be selected based on the specific dimensions of the screen, the model of the device, or other factors. Any target aspect ratio may be selected based on the desired result. For illustrative purposes, aspect ratios are considered greater than or less than one another. Such comparisons can be made by deriving values based on each number of the aspect ratio as the numerator and denominator of a fraction, respectively. For example, an aspect ratio of 4:3 has a value of 1.33, which is greater than an aspect ratio of 1:1 having a value of 1.00.

[0055] In some embodiments, the first aspect ratio and the second aspect ratio are based on a first target aspect ratio and a second target aspect ratio, respectively. The first target aspect ratio and the second target aspect ratio may be collectively referred to herein as a target aspect ratio range or a "range". The first target aspect ratio may be referred to herein as the low end of the range and the second target aspect ratio may be referred to herein as the high end of the range. For example, the target aspect ratio range may include a range from 3:4 to 4:3. Thus, the device may select a first aspect ratio that is higher than the range and a second aspect ratio that is lower than the range. Such embodiments enable the device to have more fine-grained control over each user interface layout used in each orientation.

[0056] Specifically, in response to determining that the display screen 101 is in a portrait orientation, one or more devices may cause a first user interface layout 102A to be displayed, the first user interface layout 102A including respective renderings 103 of respective ones of a plurality of streams, where each of the respective renderings each has a first aspect ratio that is greater than the first target aspect ratio or greater than the high end of the range.

[0057] Next, at operation 508, one or more computing modules may receive updated orientation data from sensors installed in the computing device. The orientation data may indicate that the display screen 101 of the computing device 100 is in a portrait orientation or a landscape orientation. As described in more detail below, Figure 7 , the sensors may include an accelerometer, a compass, or any other device for detecting the physical orientation of the device, e.g., whether the device is in a vertical position, e.g., portrait orientation, or a lateral position, e.g., landscape orientation.

[0058] Next, at operation 510, one or more computing modules may cause a second user interface layout 102B to be displayed, the second user interface layout 102B including respective renderings 103 of respective ones of a plurality of streams. Responsive to determining that the display screen 101 is in a landscape orientation, each of the respective renderings may each have a second aspect ratio that is less than a target aspect ratio. Responsive to determining that the display screen 101 is in a landscape orientation, each of the respective renderings may each have a second aspect ratio that is less than a first aspect ratio.

[0059] For example, respective renderings of the updated interface layout 102B may each be displayed using a second aspect ratio that forms a portrait dimension of 5:7, which aspect ratio is less than a target aspect ratio such as 1:1. It will be appreciated that other target aspect ratios may be selected to provide different results for each layout. Additionally, the target aspect ratio may be selected based on the specific dimensions of the screen, the model of the device, or other factors. Any target aspect ratio may be selected based on the desired result.

[0060] At operation 510, responsive to determining that the display screen 101 is in a landscape orientation, each of the respective renderings may each have a second aspect ratio that is less than a first aspect ratio. Thus, in some configurations, the aspect ratio of an image displayed on the device in a portrait orientation may be 16:9. When the device is rotated to a landscape orientation, the aspect ratio of the image displayed on the device may be less than 16:9, such as, but not limited to, 4:3, 1:1, or 5:7.

[0061] In an embodiment using a range, responsive to determining that the display screen 101 is in a landscape orientation, one or more devices may cause a second user interface layout 102B to be displayed, the second user interface layout 102B including respective renderings 103 of respective ones of a plurality of streams, wherein each of the respective renderings each has a second aspect ratio that is less than a second target aspect ratio, or a second aspect ratio that is less than the lower end of the range.

[0062] Next, at operation 512, one or more computing modules may analyze any user input for adjusting the aspect ratio of a rendering. For example, if the user resizes a specific rendering after rotating the device, the system may analyze the input to modify the target aspect ratio. Thus, the target aspect ratio may be modified over time to optimize the user interface layout for a particular user. Input data for configuring the aspect ratio of a rendering and any corresponding machine learning data may be stored on a per-user and per-device basis. Additionally, input data for configuring the aspect ratio of a rendering and any corresponding machine learning data may be stored on a per-event basis. Thus, a first target aspect ratio may be determined and stored for the portrait orientation of the device, and a second target aspect ratio may be determined and stored for the landscape orientation of the device. Each iteration of routine 500 may adjust the target aspect ratio of the device over time or switch between a single target aspect ratio and multiple target aspect ratios of the device based on the user's input pattern.

[0063] In an illustrative example, the device may start with a single target aspect ratio, e.g., a target aspect ratio of one (1:1). Then, over time, if the user adjusts the aspect ratio while the device is in a landscape orientation, e.g., the user prefers to widen each rendering, the device may switch to a mode that uses two target aspect ratios, e.g., the original target aspect ratio of the device when the device remains in a portrait orientation, and a second target aspect ratio of the device when the device remains in a landscape orientation.

[0064] In an illustrative example, when the device is in a portrait orientation, the device may receive user input to adjust a first aspect ratio. The user may adjust the aspect ratio of the device while the device remains in a portrait orientation, where the user input may increase the aspect ratio of one of the renderings. When the user increases the aspect ratio of one rendering, the device may increase the aspect ratio of each other rendering based on the user input such that each rendering (103A - 103D) has an equal aspect ratio. In response to the input, the device may generate usage data that defines an adjusted target aspect ratio based on the input used to adjust the first aspect ratio. If the user input increases the first aspect ratio of the first user interface layout, the adjusted target aspect ratio may be greater than the target aspect ratio. If the user decreases the first aspect ratio of the first user interface layout, the adjusted target aspect ratio may also be less than the target aspect ratio. Then, the device may store the adjusted target aspect ratio such that subsequent execution of the method sets the first aspect ratio to be greater than the adjusted target aspect ratio. For illustrative purposes, aspect ratios are considered greater than or less than each other. This comparison may be made by deriving values based on each number of the aspect ratio as the numerator and denominator of a fraction, respectively. For example, an aspect ratio of 4:3 has a value of 1.33, which is greater than an aspect ratio of 1:1 that has a value of 1.00.

[0065] In another illustrative example, when the device is in a landscape orientation, the device may receive user input to adjust a second aspect ratio. Then, the device may generate usage data that defines an adjusted target aspect ratio based on the input used to adjust the second aspect ratio. If the user input increases the second aspect ratio, the adjusted target aspect ratio may be greater than the target aspect ratio, or if the user input decreases the second aspect ratio, the adjusted target aspect ratio may be less than the target aspect ratio, e.g., making each rendering narrower. Similar to the example above, if the user adjusts the aspect ratio of one rendering, the device may adjust the aspect ratio of each rendering such that each rendering has the same aspect ratio based on the input. Then, the device may store the adjusted target aspect ratio such that subsequent use of the device sets the second aspect ratio to a value less than the adjusted target aspect ratio.

[0066] It should be appreciated that the above subject matter may be implemented as a computer-controlled apparatus, a computer process, a computing system, or as an article of manufacture such as a computer-readable storage medium. Operations of example methods are illustrated in and summarized with reference to these blocks. The methods are depicted as a logical flow of blocks, each of which may represent one or more operations that may be implemented in hardware, software, or a combination thereof. In a software context, the operations represent computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, enable the one or more processors to perform the recited operations.

[0067] Generally, computer-executable instructions include routines, programs, objects, modules, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order of description of the operations is not intended to be construed as a limitation, and any number of the described operations may be executed in any order, combined in any order, subdivided into multiple sub-operations, and / or executed in parallel to implement the described process. The described process may be executed by resources associated with one or more devices, such as one or more internal or external CPUs or GPUs, and / or one or more hardware logics, such as a field-programmable gate array (“FPGA”), a digital signal processor (“DSP”), or other types of accelerators.

[0068] All of the above methods and processes may be embodied in software code modules executed by one or more general-purpose computers or processors and be fully automated by the software code modules executed by one or more general-purpose computers or processors. The code modules may be stored in any type of computer-readable storage medium or other computer storage device, such as those described below. Some or all of the methods may alternatively be embodied in dedicated computer hardware, such as those described below.

[0069] Any routine descriptions, elements, or blocks in the flowcharts described herein and / or depicted in the figures should be understood as potentially representing modules, segments, or portions of code that include one or more executable instructions for implementing specific logical functions or elements in the routine. Alternative implementations are included within the scope of the examples described herein, where elements or functions may be deleted, or executed out of order from that shown or discussed, including substantially synchronously or in reverse order, depending on the functionality involved, as would be understood by those skilled in the art.

[0070] Figure 7 FIG. 600 is a diagram of an example environment 600 in which system 602 may implement the techniques disclosed herein. In some implementations, system 602 may be used to collect, analyze, and share data defining one or more objects that are presented to a user of communication session 604.

[0071] As shown, a communication session 603 can be implemented between multiple client computing devices 606(1) to 606(N) (where N is a number with a value of 2 or greater) associated with or as part of a system 602. The client computing devices 606(1) to 606(N) enable users (also referred to as individuals) to participate in the communication session 603.

[0072] In this example, the communication session 603 is hosted by the system 602 over one or more networks 608. That is, the system 602 can provide services that enable users of the client computing devices 606(1) to 606(N) to participate in the communication session 603 (e.g., by live viewing and / or recorded viewing). Thus, the "participants" in the communication session 603 can include users and / or client computing devices (e.g., multiple users can participate in the communication session by using a single client computing device in a room), and each of the users and / or client computing devices can communicate with other participants. Alternatively, the communication session 603 can be hosted by one of the client computing devices 606(1) to 606(N) using peer-to-peer technology. The system 602 can also host chat conversations and other team collaboration functions (e.g., as part of an application suite).

[0073] In some embodiments, such chat conversations and other team collaboration functions are considered separate external communication sessions from the communication session 603. The computing system 602 that collects participant data in the communication session 603 may be able to link to such external communication sessions. Thus, the system can receive information capable of connecting to such external communication sessions, such as dates, times, session details, etc. In one example, a chat conversation can be based on the communication session 603. Additionally, the system 602 can host the communication session 603, which includes at least multiple participants co-located at a meeting location (e.g., a meeting room or auditorium) or located at different locations.

[0074] In the example described herein, the client computing devices 606(1) to 606(N) participating in the communication session 603 are configured to receive and render communication data for display on a user interface of a display screen. The communication data can include a collection of various instances or streams of live content and / or recorded content. The collection of various instances or streams of live content and / or recorded content can be provided by one or more cameras such as a video camera. For example, individual streams of live or recorded content can include media data associated with a video feed provided by the camera (e.g., audio and visual data capturing the appearance and speech of users participating in the communication session). In some embodiments, the video feed can include such audio and visual data, one or more still images, and / or one or more avatars. The one or more still images can also include one or more avatars.

[0075] Another example of a separate stream of live or recorded content may include media data that includes an avatar of a user participating in a communication session and audio data that captures the user's voice. Yet another example of a separate stream of live or recorded content may include media data that includes a file displayed on a display screen and audio data that captures the user's voice. Thus, the various live or recorded content streams within the communication data enable remote conferencing among a group of people and content sharing within that group of people. In some embodiments, the various live or recorded content streams within the communication data may originate from a plurality of co-located cameras located in a space such as a room to live record or stream presentation content that includes one or more individuals presenting and one or more individuals consuming the presented content.

[0076] Participants or attendees can view the content of communication session 603 in real time as the event occurs or, alternatively, view the content of communication session 603 at a later time after the event via a recording. In the examples described herein, the client computing devices 606(1) through 606(N) participating in communication session 603 are configured to receive and render communication data for display on a user interface of a display screen. The communication data may include a collection of various instances or streams of live and / or recorded content. For example, a separate content stream may include media data associated with a video feed (e.g., audio and visual data that captures the appearance and voice of a user participating in a communication session). Another example of a separate content stream may include media data that includes an avatar of a user participating in a meeting session and audio data that captures the user's voice. Yet another example of a separate content stream may include media data that includes a content item displayed on a display screen and / or audio data that captures the user's voice. Thus, the various content streams within the communication data enable a meeting or broadcast presentation to be facilitated among a group of people dispersed at remote locations.

[0077] A participant or attendee of a communication session is a person within the range of a camera or other image and / or audio capture device such that the person's actions and / or sounds that occur while the person views and / or listens to the content shared via the communication session can be captured (e.g., recorded). For example, a participant may be sitting in a crowd watching shared content at a broadcast location where a stage presentation is occurring. Or, a participant may be sitting in an office conference room viewing shared content of a communication session with other colleagues via a display screen. Further still, a participant may be sitting or standing in front of a personal device (e.g., a tablet, smartphone, computer, etc.) viewing shared content of a communication session alone in their office or at home.

[0078] Figure 6The system 602 includes a device 610. The device 610 and / or other components of the system 602 may include distributed computing resources that communicate with each other and / or with client computing devices 606(1) to 606(N) via one or more networks 608. In some examples, the system 602 may be a stand-alone system tasked with managing various aspects of one or more communication sessions, such as communication session 603. As an example, the system 602 may be managed by entities such as SLACK, WEBEX, GOTOMEETING, GOOGLE HANGOUTS, etc.

[0079] The network 608 may include, for example, a public network such as the Internet, a private network such as an institutional and / or personal intranet, or some combination of a private network and a public network. The network 608 may also include any type of wired and / or wireless network, including but not limited to a local area network (“LAN”), a wide area network (“WAN”), a satellite network, a wired network, a Wi-Fi network, a WiMax network, a mobile communication network (e.g., 3G, 4G, etc.), or any combination thereof. The network 608 may utilize communication protocols, including packet-based and / or datagram-based protocols such as the Internet Protocol (“IP”), the Transmission Control Protocol (“TCP”), the User Datagram Protocol (“UDP”), or other types of protocols. In addition, the network 608 may also include multiple devices that facilitate network communication and / or form the network hardware foundation, such as switches, routers, gateways, access points, firewalls, base stations, repeaters, backbone devices, and the like.

[0080] In some examples, the network 608 may also include devices capable of connecting to a wireless network, such as a wireless access point (“WAP”). Examples support the connectivity of WAPs that send and receive data at various electromagnetic frequencies (e.g., radio frequencies), including WAPs that support Institute of Electrical and Electronics Engineers (“IEEE”) 802.11 standards (e.g., 802.11g, 802.11h, 802.11ac, etc.) and other standards.

[0081] In various examples, the device 610 may include one or more computing devices that operate in a cluster or other grouped configuration to share resources, balance loads, improve performance, provide failover support or redundancy, or for other purposes. For example, the device 610 may belong to various device categories, such as traditional server-type devices, desktop computer-type devices, and / or mobile-type devices. Thus, although shown as a single type of device or a server-type device, the device 610 may include multiple device types and is not limited to a specific type of device. The device 610 may represent, but is not limited to, a server computer, a desktop computer, a network server computer, a personal computer, a mobile computer, a laptop computer, a tablet computer, or any other type of computing device.

[0082] The client computing devices (e.g., one of client computing devices 606(1) to 606(N), each of which is also referred to herein as a "data processing system") can belong to various categories of devices that can be the same as or different from device 610, such as traditional client-type devices, desktop computer-type devices, mobile-type devices, dedicated-type devices, embedded devices, and / or wearable devices. Thus, the client computing devices can include, but are not limited to, desktop computers, game consoles and / or gaming devices, tablet computers, personal data assistants ("PDAs"), mobile phone / tablet hybrids, laptop computers, telecommunications devices, computer navigation client computing devices, such as satellite-based navigation systems including global positioning system ("GPS") devices, wearable devices, virtual reality ("VR") devices, augmented reality ("AR") devices, implanted computing devices, automotive computers, network-enabled televisions, thin clients, terminals, Internet of Things ("IoT") devices, workstations, media players, personal video recorders ("PVRs"), set-top boxes, cameras, integrated components for inclusion in computing devices (e.g., peripherals), appliances, or any other type of computing device. Additionally, the client computing devices can include combinations of the client computing device examples listed above, such as a desktop computer-type device or a mobile-type device combined with a wearable device, etc.

[0083] The client computing devices 606(1) to 606(N) of various categories and device types can represent any type of computing device having one or more data processing units 692 operably connected to a computer-readable medium 694, for example, via a bus 616, which in some cases can include one or more of a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and any kind of local, peripheral, and / or independent bus.

[0084] The executable instructions stored on the computer-readable medium 694 can include, for example, an operating system 619, sensors 620, modules 622, and other modules, programs, or applications that can be loaded and executed by the data processing unit 692. The sensors 620 can be accelerometers, compasses, or any other solid-state devices that can detect the orientation, acceleration, or position of the device.

[0085] The client computing devices 606(1) through 606(N) may also include one or more interfaces 624 to enable communication between the client computing devices 606(1) through 606(N) and other network devices (such as device 610) over the network 608. Such network interfaces 624 may include one or more network interface controllers (NICs) or other types of transceiver devices to send and receive communications and / or data over the network. Additionally, the client computing devices 606(1) through 606(N) may include input / output (“I / O”) interfaces (devices) 626 that allow for communication with input / output devices, such as user input devices including peripheral input devices (e.g., game controllers, keyboards, mice, pens, voice input devices such as microphones, cameras for acquiring and providing video feeds and / or still images, touch input devices, gesture input devices, etc.) and / or output devices including peripheral output devices (e.g., displays, printers, audio speakers, tactile output devices, etc.). Figure 6 Illustrated is that the client computing device 606(1) is connected in some manner to a display device (e.g., display screen 629(N)), which may display a UI in accordance with the techniques described herein.

[0086] In Figure 7 the example environment 600, the client computing devices 606(1) through 606(N) may use their respective client modules 620 to connect to each other and / or to other external devices to participate in a communication session 603 or to contribute activities to the collaborative environment. For example, a first user may use the client computing device 606(1) to communicate with a second user of another client computing device 606(2). When the client module 620 is executed, the users may share data, which may cause the client computing device 606(1) to connect to the system 602 and / or other client computing devices 606(2) through 606(N) over the network 608.

[0087] The client computing devices 606(1) through 606(N) may use their respective modules 622 to generate participant profiles ( Figure 7 not shown in the figure) and provide the participant profiles to other client computing devices and / or the device 610 of the system 602. The participant profile may include the identity of a user or group of users (e.g., name, unique identifier (“ID”), etc.), user data such as personal data, machine data such as location (e.g., IP address, room in a building, etc.), and technical capabilities, among other things. The participant profile may be used to register participants for a communication session.

[0088] As Figure 7As shown, the device 610 of the system 602 includes a server module 630 and an output module 632. In this example, the server module 630 is configured to receive media streams 634(1) to 634(N) from separate client computing devices such as client computing devices 606(1) to 606(N). As described above, the media streams can include video feeds (e.g., audio and video data associated with a user), audio data to be output along with the rendering of the user's avatar (e.g., a pure audio experience where the user's video data is not transmitted), text data (e.g., text messages), file data, and / or screen sharing data (e.g., documents, slides, images, videos displayed on a display screen), and so on. Thus, the server module 630 is configured to receive a collection of various media streams 634(1) to 634(N) (this collection is referred to herein as "media data 634") during a live viewing of the communication session 603. In some scenarios, not all client computing devices participating in the communication session 603 provide media streams. For example, a client computing device may be just a consumption or "listening" device, so it only receives content associated with the communication session 603 but does not provide any content to the communication session 603.

[0089] In various examples, the server module 630 can select aspects of the media streams 634 to share with each of the participating client computing devices 606(1) to 606(N). Thus, the server module 630 can be configured to generate session data 636 based on the streams 634 and / or pass the session data 636 to the output module 632. Then, the output module 632 can transmit communication data 639 to the client computing devices (e.g., client computing devices 606(1) to 606(3) participate in the live viewing of the communication session). The communication data 639 can include video, audio, and / or other content data provided by the output module 632 based on the content 650 associated with the output module 632 and based on the received session data 636. The content 650 can include the streams 634 or other shared data, such as image files, spreadsheet files, slides, documents, etc. The streams 634 can include video components depicting images captured by the I / O devices 626 on each client computer.

[0090] As shown, the output module 632 sends communication data 639(1) to the client computing device 606(1), sends communication data 639(2) to the client computing device 606(2), and sends communication data 639(3) to the client computing device 606(3), and so on. The communication data 639 sent to the client computing devices can be the same or can be different (e.g., the positioning of the content stream within the user interface can be different from one device to the next).

[0091] In various embodiments, device 610 and / or client module 620 may include module 622, which may also be referred to herein as the GUI module. The GUI module 622 may be configured to analyze communication data 639 for transmission to one or more client computing devices 606. Specifically, the UI module 622 at the device 610 and / or the client computing device 606 may analyze the communication data 639 to determine an appropriate manner for displaying video, images, and / or content on the display screen 629 of the associated client computing device 606. In some implementations, the GUI module 622 may provide video, images, and / or content to a presentation GUI 646 that is rendered on the display screen 629 of the associated client computing device 606. The GUI module 622 may cause the presentation GUI 646 to be rendered on the display screen 629. The presentation GUI 646 may include video, images, and / or content analyzed by the GUI module 622.

[0092] In some embodiments, the presentation GUI 646 may include multiple sections or grids that may render or include video, images, and / or content for display on the display screen 629. For example, a first section of the presentation GUI 646 may include a video feed of a presenter or individual, and a second section of the presentation GUI 646 may include a video feed of an individual consuming meeting information provided by the presenter or individual. The GUI module 622 may populate the first and second sections of the presentation GUI 646 in a manner that appropriately mimics the environmental experience that the presenter and individuals may be sharing.

[0093] In some embodiments, the GUI module 622 may magnify or provide a zoomed-in view of an individual represented by a video feed to highlight the individual's reaction to the presenter, such as facial features. In some embodiments, the presentation GUI 646 may include video feeds of multiple participants associated with a meeting, such as a general communication session. In other embodiments, the presentation GUI 646 may be associated with a channel such as a chat channel, an enterprise team channel, etc. Thus, the presentation GUI 646 may be associated with an external communication session different from a general communication session.

[0094] Figure 8 A diagram illustrates example components of an example device 700 (also referred to herein as a "computing device") that is configured to generate data for some of the user interfaces disclosed herein. The device 700 may generate data that may include one or more sections that may render or include video, images, virtual objects, and / or content for display on the display screen 629. The device 700 may represent one of the devices described herein. Additionally or alternatively, the device 700 may represent one of the client computing devices 606.

[0095] As shown in the figure, device 700 includes one or more data processing units 702, a computer-readable medium 704, and a communication interface 706. The components of device 700 are operatively connected, for example, via a bus 709, which may include a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and one or more of any kind of local, peripheral, and / or independent bus.

[0096] As used herein, a data processing unit such as data processing unit 702 and / or data processing unit 692 may represent, for example, a CPU-type data processing unit, a GPU-type data processing unit, a field programmable gate array (“FPGA”), another type of DSP, or other hardware logic components, which in some cases may be driven by a CPU. By way of example, and not limitation, illustrative types of hardware logic components that may be used include application specific integrated circuits (“ASICs”), application specific standard products (“ASSPs”), system-on-a-chip systems (“SOCs”), complex programmable logic devices (“CPLDs”), and the like.

[0097] As used herein, a computer-readable medium such as computer-readable medium 704 and computer-readable medium 694 may store instructions executable by a data processing unit. The computer-readable medium may also store instructions executable by an external data processing unit such as an external CPU, an external GPU, etc. and / or by an accelerator such as an FPGA-type accelerator, a DSP-type accelerator, or any other internal or external accelerator. In various examples, at least one CPU, GPU, and / or accelerator is incorporated into the computing device, while in some examples, one or more of the CPU, GPU, and / or accelerator are external to the computing device.

[0098] A computer-readable medium, also referred to herein as a computer-readable medium, may include computer storage media and / or communication media. Computer storage media may include volatile memory, non-volatile memory, and / or other persistent and / or auxiliary computer storage media, one or more of removable and non-removable computer storage media, implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Thus, computer storage media includes media in a tangible and / or physical form included in a device, and / or hardware components that are part of or external to a device, including but not limited to random access memory (“RAM”), static random access memory (“SRAM”), dynamic random access memory (“DRAM”), phase change memory (“PCM”), read-only memory (“ROM”), erasable programmable read-only memory (“EPROM”), electrically erasable programmable read-only memory (“EEPROM”), flash memory, compact disc read-only memory (“CD-ROM”), digital versatile disc (“DVD”), optical card, or other optical storage media, magnetic cassette, tape, disk storage, magnetic card, or other magnetic storage device or media, solid-state storage device, storage array, network-attached storage, storage area network, hosted computer storage, or any other storage memory, storage device, and / or storage media that can be used to store and maintain information for access by a computing device.

[0099] In contrast to computer storage media, communication media may embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transmission mechanism. As defined herein, computer storage media does not include communication media. That is, computer storage media itself does not include communication media consisting solely of a modulated data signal, a carrier wave, or a propagated signal.

[0100] The communication interface 706 may represent, for example, a network interface controller (“NIC”) or other type of transceiver device to send and receive communications over a network. Additionally, the communication interface 706 may include one or more cameras and / or audio devices 722 to enable the generation of video feeds and / or still images, etc.

[0101] In the example shown, the computer-readable medium 704 includes a data repository 708. In some examples, the data repository 708 includes data storage, such as a database, a data warehouse, or other types of structured or unstructured data storage. In some examples, the data repository 708 includes a corpus and / or a relational database having one or more tables, indexes, stored procedures, etc., to enable data access including, for example, one or more hypertext markup language (“HTML”) tables, resource description framework (“RDF”) tables, web ontology language (“OWL”) tables, and / or extensible markup language (“XML”) tables.

[0102] The data repository 708 can store data for operations of processes, applications, components, and / or modules that are stored in the computer-readable medium 704 and / or executed by the data processing unit 702 and / or the accelerator. For example, in some examples, the data repository 708 can store session data 710, profile data 712 (e.g., associated with a participant profile), and / or other data. The session data 710 can include the total number of participants (e.g., users and / or client computing devices) in a communication session, the activities that occur in the communication session, a list of invitees for the communication session, and / or other data related to when and how the communication session is conducted or hosted. The data repository 708 can also include content data 714, such as content including video, audio, or other content for rendering and displaying on one or more display screens 629.

[0103] Alternatively, some or all of the above data can be stored on a separate memory 716 on one or more of the data processing units 702, such as memories on a CPU-type processor, a GPU-type processor, an FPGA-type accelerator, a DSP-type accelerator, and / or other accelerators. In this example, the computer-readable medium 704 also includes an operating system 718 and an application programming interface 710 (API) configured to expose the functions and data of the device 700 to other devices. Additionally, the computer-readable medium 704 includes one or more modules, such as a server module 730, an output module 732, and a GUI rendering module 740, although the number of the illustrated modules is only an example and the number can vary higher or lower. That is, the functions described in connection with the illustrated modules can be performed by a smaller number of modules or a larger number of modules on one device, or distributed across multiple devices.

[0104] The disclosure presented herein also covers the subject matter set forth in the following clauses.

[0105] Clause A: A method performed by a computing system 100, the method comprising: receiving a plurality of streams, each of the plurality of streams including a video component; receiving orientation data from a sensor installed to the computing system 100, the orientation data indicating that a display screen 101 of the computing system 100 is in a portrait orientation; in response to determining that the display screen 101 is in the portrait orientation, causing a first user interface layout 102A to be displayed that includes respective renderings 103 of each of the plurality of streams, wherein each of the respective renderings has a first aspect ratio greater than a target aspect ratio; receiving additional orientation data from the sensor, the additional orientation data indicating that the display screen has been switched from the portrait orientation to a landscape orientation; and in response to determining that the display screen 101 has been switched from the portrait orientation to the landscape orientation, causing a display to be switched from the first user interface layout 102A to a second user interface layout 102B that includes updated renderings 103 of each stream, wherein the updated renderings 103 each have a second aspect ratio less than the target aspect ratio.

[0106] Clause B: The method of Clause A, further comprising: analyzing the plurality of streams to determine that at least one of the plurality of streams depicts a threshold number of people; analyzing at least one of the streams depicting the threshold number of people to identify at least one person associated with a threshold activity level; and determining a scaling factor for generating a modified rendering of the at least one stream, the scaling factor of the modified rendering being configured to shift the focus to at least one person associated with the threshold activity level.

[0107] Clause C: The method of Clauses A and B, further comprising: analyzing the plurality of streams to determine that at least one of the plurality of streams depicts a threshold number of people; and selecting a fixed aspect ratio for the at least one stream, wherein the fixed aspect ratio encompasses the association with the second aspect ratio and the second aspect ratio, and wherein the fixed aspect ratio is used to display the rendering of the at least one stream when the display screen is in the portrait orientation or the landscape orientation.

[0108] Clause D: The method of Clauses A through C, further comprising: analyzing the plurality of streams to determine the number of people depicted in the video component of each stream; and configuring the first user interface layout or the second user interface layout to sort the respective renderings of each stream based on the number of people depicted in the video component of each stream.

[0109] Clause E: The method of Clauses A through D, further comprising: analyzing the plurality of streams to determine the number of people depicted in the video component of each stream; configuring the first user interface arrangement or the second user interface arrangement to position the rendering of the first stream at one end of a rendering sequence of the plurality of streams and to position the rendering of the second stream at a second position in the rendering sequence, wherein the number of people depicted in the first stream is greater than the number of people depicted in the second stream; analyzing the plurality of streams to determine whether the number of people depicted in the first stream is less than the number of people depicted in the second stream; and in response to determining that the number of people depicted in the first stream is less than the number of people depicted in the second stream, configuring the first user interface arrangement or the second user interface arrangement to move the rendering of the second stream to one end of the rendering sequence and to position the rendering of the first stream at the second position in the rendering sequence.

[0110] Clause F: The method of Clauses A through E, wherein the target aspect ratio is selected based on one or more dimensions of a display device in communication with the computing system, and wherein the target aspect ratio includes a range having a low ratio and a high ratio, wherein the first aspect ratio is greater than the high ratio of the range, and the second aspect ratio is less than the low ratio of the range.

[0111] Clause G: The method of Clauses A through F, further comprising: receiving user input to adjust the first aspect ratio when the device is in a portrait orientation; generating usage data defining an adjusted target aspect ratio based on the input for adjusting the first aspect ratio, wherein if the user input increases the first aspect ratio, the adjusted target aspect ratio is greater than the target aspect ratio; and storing the adjusted target aspect ratio such that subsequent execution of the method sets the first aspect ratio to be greater than the adjusted target aspect ratio.

[0112] Clause H: The method of Clauses A through G, further comprising: receiving user input to adjust the second aspect ratio when the device is in a landscape orientation; generating usage data defining an adjusted target aspect ratio based on the input for adjusting the second aspect ratio, wherein if the user input increases the second aspect ratio, the adjusted target aspect ratio is greater than the target aspect ratio; and storing the adjusted target aspect ratio such that subsequent execution of the method sets the second aspect ratio to a value less than the adjusted target aspect ratio.

[0113] Clause I: System 100 includes: one or more processing units 692; and a computer-readable medium 692 encoded with computer-executable instructions to cause the one or more processing units 692 to: receive a plurality of streams, each of the plurality of streams including a video component; receive orientation data from a sensor installed in the computing system 100, the orientation data indicating that the display screen 101 of the computing system 100 is in a landscape orientation; in response to determining that the display screen 101 is in a landscape orientation, cause a first user interface layout 102B to be displayed that includes respective renderings 103' of each of the plurality of streams, where each rendering is displayed using a first aspect ratio; receive additional orientation data from the sensor, the additional orientation data indicating that the display screen 101 has been switched from a landscape orientation to a portrait orientation; and in response to determining that the display screen 101 has been switched from a landscape orientation to a portrait orientation, cause a display to be switched from the first user interface layout 102B to a second user interface layout 102A that includes updated renderings 103 of each of the streams, where the updated renderings 103 are each displayed using a second aspect ratio that is greater than the first aspect ratio.

[0114] Clause J: The system of Clause I, where the instructions further cause the one or more processing units to: analyze the plurality of streams to determine that at least one of the plurality of streams depicts a threshold number of people; analyze at least one of the streams depicting the threshold number of people to identify at least one person associated with a threshold activity level; and determine a scaling factor for generating a modified rendering of the at least one stream, the scaling factor of the modified rendering being configured to shift the focus to the at least one person associated with the threshold activity level.

[0115] Clause K: The system of Clauses I and J, where the instructions further cause the one or more processing units to: analyze the plurality of streams to determine that at least one of the plurality of streams depicts a threshold number of people; and select a fixed aspect ratio for the at least one stream, where the fixed aspect ratio overrides the association with and the second aspect ratio, and where the fixed aspect ratio is used to display the rendering of the at least one stream when the display screen is in a portrait orientation or a landscape orientation.

[0116] Clause L: The system of Clauses I through K, where the instructions further cause the one or more processing units to: analyze the plurality of streams to determine the number of people depicted in the video component of each stream; and configure the first user interface layout or the second user interface layout to sort the respective renderings of each stream based on the number of people depicted in the video component of each stream.

[0117] Clause M: A system of Clauses I through L, wherein the instructions further cause one or more processing units to: analyze the plurality of streams to determine the number of people depicted in the video component of each stream; configure the first user interface arrangement or the second user interface arrangement to position the rendering of the first stream at one end of the rendering sequence of the plurality of streams and to position the rendering of the second stream at a second position in the rendering sequence, wherein the number of people depicted in the first stream is greater than the number of people depicted in the second stream; analyze the plurality of streams to determine whether the number of people depicted in the first stream is less than the number of people depicted in the second stream; and in response to determining that the number of people depicted in the first stream is less than the number of people depicted in the second stream, configure the first user interface arrangement or the second user interface arrangement to move the rendering of the second stream to one end of the rendering sequence and to position the rendering of the first stream at the second position in the rendering sequence.

[0118] Clause N: A system 110, comprising: means for receiving a plurality of streams, each of the plurality of streams including a video component; means for receiving orientation data from a sensor mounted to the computing system 100, the orientation data indicating that the display screen 101 of the computing system 100 is in a portrait orientation; means for causing a first user interface arrangement 102A to be displayed that includes respective renderings 103 of each of the plurality of streams in response to determining that the display screen 101 is in a portrait orientation, wherein each of the respective renderings has a first aspect ratio greater than a target aspect ratio; means for receiving additional orientation data from the sensor, the additional orientation data indicating that the display screen 101 has been switched from a portrait orientation to a landscape orientation; and means for causing a display to be switched from the first user interface arrangement 102A to a second user interface arrangement 102B that includes updated renderings 103' of each of the streams in response to determining that the display screen 101 has been switched from a portrait orientation to a landscape orientation, wherein each of the updated renderings has a second aspect ratio less than the target aspect ratio.

[0119] Clause O: The system of Clause N, further comprising: means for analyzing the plurality of streams to determine that at least one of the plurality of streams depicts a threshold number of people; means for analyzing at least one stream depicting the threshold number of people to identify at least one person associated with a threshold activity level; and means for determining a scaling factor for generating a modified rendering of the at least one stream, the scaling factor of the modified rendering being configured to shift the focus to the at least one person associated with the threshold activity level.

[0120] Clause P: The system of Clauses N and O, further comprising: means for analyzing the plurality of streams to determine that at least one of the plurality of streams depicts a threshold number of people; and means for selecting a fixed aspect ratio for the at least one stream, wherein the fixed aspect ratio encompasses the association with the second aspect ratio and the second aspect ratio, and wherein when the display screen is in a portrait orientation or a landscape orientation, the fixed aspect ratio is used to display the rendering of the at least one stream.

[0121] Clause Q: The system of Clauses N to P, further comprising: means for analyzing the plurality of streams to determine the number of people depicted in the video component of each stream; and means for configuring the first user interface layout or the second user interface layout to sort the respective renderings of each stream based on the number of people depicted in the video component of each stream.

[0122] Clause R: The system of Clauses N to Q, further comprising: means for analyzing the plurality of streams to determine the number of people depicted in the video component of each stream; means for configuring the first user interface layout or the second user interface layout to position the rendering of a first stream at one end of a rendering sequence of the plurality of streams and to position the rendering of a second stream at a second position in the rendering sequence, wherein the number of people depicted in the first stream is greater than the number of people depicted in the second stream; means for analyzing the plurality of streams to determine whether the number of people depicted in the first stream is less than the number of people depicted in the second stream; and means for configuring the first user interface layout or the second user interface layout to move the rendering of the second stream to one end of the rendering sequence and to position the rendering of the first stream at the second position in the rendering sequence in response to determining that the number of people depicted in the first stream is less than the number of people depicted in the second stream.

[0123] Clause S: The system of Clauses N to R, further comprising: means for receiving user input to adjust the first aspect ratio when the device is in a portrait orientation; means for generating usage data defining an adjusted target aspect ratio based on the input for adjusting the first aspect ratio, wherein if the user input increases the first aspect ratio, the adjusted target aspect ratio is greater than the target aspect ratio; and means for storing the adjusted target aspect ratio such that subsequent execution of the method sets the first aspect ratio to be greater than the adjusted target aspect ratio.

[0124] Clause T: The system of Clauses N to S further includes: means for receiving user input to adjust the second aspect ratio when the device is in a landscape orientation; means for generating usage data defining an adjusted target aspect ratio based on the input for adjusting the second aspect ratio, where if the user input increases the second aspect ratio, the adjusted target aspect ratio is greater than the target aspect ratio; and means for storing the adjusted target aspect ratio such that subsequent execution of the method sets the second aspect ratio to a value less than the adjusted target aspect ratio.

[0125] It should also be appreciated that many changes and modifications can be made to the above examples, and the elements thereof should be understood to be in other acceptable examples. All such modifications and changes are intended to be included within the scope of the present disclosure and are protected by the following claims.

[0126] Finally, although the various configurations have been described in language specific to structural features and / or method acts, it should be understood that the subject matter defined in the appended representations is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claimed subject matter.

Claims

1. A method performed by a computing system, the method comprising: Receiving a plurality of streams, each stream of the plurality of streams including a video component; Receiving orientation data from a sensor installed in the computing system, the orientation data indicating that a display screen of the computing system is in a portrait orientation; In response to determining that the display screen is in the portrait orientation, causing a first user interface layout including respective renderings of each stream of the plurality of streams to be displayed, wherein each of the respective renderings has a first aspect ratio greater than a target aspect ratio; Receiving additional orientation data from the sensor, the additional orientation data indicating that the display screen is switched from the portrait orientation to a landscape orientation; In response to determining that the display screen is switched from the portrait orientation to the landscape orientation, causing a display to be switched from the first user interface layout to a second user interface layout including updated renderings of the respective streams, wherein each of the updated renderings has a second aspect ratio less than the target aspect ratio; Analyzing the plurality of streams to determine that at least one stream of the plurality of streams depicts a threshold number of people; and Selecting a fixed aspect ratio for the at least one stream, wherein the fixed aspect ratio encompasses the first aspect ratio and the second aspect ratio, and wherein the rendering of the at least one stream is displayed using the fixed aspect ratio when the display screen is in the portrait orientation or the landscape orientation.

2. The method of claim 1, further comprising: Analyzing the at least one stream depicting the threshold number of people to identify at least one person associated with a threshold activity level; And Determining a scaling factor for generating a modified rendering of the at least one stream, the scaling factor of the modified rendering being configured to shift the focus to the at least one person associated with the threshold activity level.

3. The method of claim 1, further comprising: Analyzing the plurality of streams to determine the number of people depicted in the video component of each stream; And Configuring the first user interface layout or the second user interface layout to sort the respective renderings of each stream based on the number of people depicted in the video component of each stream.

4. The method of claim 1, further comprising: Analyzing the plurality of streams to determine the number of people depicted in the video component of each stream; Configuring the first user interface layout or the second user interface layout to position the rendering of a first stream at one end of a rendering sequence of the plurality of streams and to position the rendering of a second stream at a second position in the rendering sequence, wherein the number of people depicted in the first stream is greater than the number of people depicted in the second stream; Analyzing the plurality of streams to determine whether the number of people depicted in the first stream is less than the number of people depicted in the second stream; And In response to determining that the number of people depicted in the first stream is less than the number of people depicted in the second stream, configuring the first user interface layout or the second user interface layout to move the rendering of the second stream to one end of the rendering sequence and to position the rendering of the first stream at the second position in the rendering sequence.

5. The method according to claim 1, wherein, The target aspect ratio is selected based on one or more dimensions of a display device in communication with the computing system, and wherein the target aspect ratio includes a range having a low ratio and a high ratio, wherein the first aspect ratio is greater than the high ratio of the range, and the second aspect ratio is less than the low ratio of the range.

6. The method of claim 1, further comprising: Receiving user input for adjusting the first aspect ratio when the computing system is in the portrait orientation; Generating usage data defining an adjusted target aspect ratio that is based on the input for adjusting the first aspect ratio, wherein if the user input increases the first aspect ratio, the adjusted target aspect ratio is greater than the target aspect ratio; And Storing the adjusted target aspect ratio such that subsequent execution of the method sets the first aspect ratio to be greater than the adjusted target aspect ratio.

7. The method of claim 1, further comprising: Receiving user input for adjusting the second aspect ratio when the computing system is in the landscape orientation; Generating usage data defining an adjusted target aspect ratio that is based on the input for adjusting the second aspect ratio, wherein if the user input increases the second aspect ratio, the adjusted target aspect ratio is greater than the target aspect ratio; And Storing the adjusted target aspect ratio such that subsequent execution of the method sets the second aspect ratio to a value less than the adjusted target aspect ratio.

8. A computing system, comprising: One or more processing units; And A computer-readable medium encoded with computer-executable instructions to cause the one or more processing units to: Receive a plurality of streams, each of the plurality of streams including a video component; Receive orientation data from a sensor installed in the computing system, the orientation data indicating that a display screen of the computing system is in a landscape orientation; In response to determining that the display screen is in the landscape orientation, cause a first user interface layout including respective renderings of each of the plurality of streams to be displayed, wherein each of the respective renderings is displayed using a first aspect ratio; Receive additional orientation data from the sensor, the additional orientation data indicating that the display screen is transitioning from the landscape orientation to a portrait orientation; In response to determining that the display screen is transitioning from the landscape orientation to the portrait orientation, cause a display to transition from the first user interface layout to a second user interface layout including updated renderings of each of the streams, wherein each of the updated renderings is displayed using a second aspect ratio greater than the first aspect ratio; Analyze the plurality of streams to determine that at least one of the plurality of streams depicts a threshold number of people; and Select a fixed aspect ratio for the at least one stream, wherein the fixed aspect ratio encompasses the first aspect ratio and the second aspect ratio, and wherein the rendering of the at least one stream is displayed using the fixed aspect ratio when the display screen is in the portrait orientation or the landscape orientation.

9. The system according to claim 8, wherein, The instructions further cause the one or more processing units to: Analyze at least one stream of the people depicting the threshold number to identify at least one person associated with a threshold activity level; And Determine a scaling factor for generating a modified rendering of the at least one stream, the scaling factor of the modified rendering being configured to shift the focus to the at least one person associated with the threshold activity level.

10. The system according to claim 8, wherein, The instructions further cause the one or more processing units to: Analyze the multiple streams to determine the number of people depicted in the video component of each stream; And Configure the first user interface layout or the second user interface layout to sort the respective renderings of each stream based on the number of people depicted in the video component of each stream.

11. The system according to claim 8, wherein, The instructions further cause the one or more processing units to: Analyze the multiple streams to determine the number of people depicted in the video component of each stream; Configure the first user interface layout or the second user interface layout to position the rendering of the first stream at one end of the rendering sequence of the multiple streams and position the rendering of the second stream at a second position in the rendering sequence, wherein the number of people depicted in the first stream is greater than the number of people depicted in the second stream; Analyze the multiple streams to determine whether the number of people depicted in the first stream is less than the number of people depicted in the second stream; and In response to determining that the number of people depicted in the first stream is less than the number of people depicted in the second stream, configure the first user interface layout or the second user interface layout to move the rendering of the second stream to one end of the rendering sequence and position the rendering of the first stream at the second position in the rendering sequence.

Images