Automation of visual indicators for distinguishing active speakers of user displayed as three-dimensional representation

Abstract

The disclosed technology provides a system to automate visual indicators to display active speakers of a communication session displayed as a 3D representation. Some participants of a communication session may be displayed in a user interface using 3D representations (e.g., avatars) that are each located within a 3D environment. The user interface may also include multiple renders of 2D images of other participants displayed in the gallery (e.g., display areas designated for active speakers). When a user displayed as a 3D representation begins speaking, the system may detect speaker "activity via detection of an audio signal from the user device. In response to the detection, the system may then automatically add the user's supplemental image to the gallery. The supplemental image may help the viewer navigate through a complex user interface arrangement that displays a large number of avatars.

Description

Background Art

[0001] The use of virtual reality environments for online conference applications is becoming ubiquitous. Participants in online conferences now meet in three-dimensional (3D) virtual environments and share content within these virtual environments. Although there are many benefits compared to other forms of collaboration, using 3D environments to share content brings many disadvantages.

[0002] One of the main problems of using a virtual reality environment for online conference applications is that there may be scenarios where it may be difficult for participants in the meeting to identify relevant user activities. Some systems that support 3D environments may allow many users to participate. Some 3D environments allow hundreds of users. For certain types of events, such as meetings, it may be difficult for users to identify specific conversations and specific people involved in the activities of interest. This problem may be exacerbated when the user interface arrangement includes a combination of 2D and 3D renderings. In such an arrangement, the display of the 3D environment may only use a portion of the display screen, thereby reducing the size of the rendering of the 3D environment and making it more difficult to identify relevant user activities within the display of the 3D environment. For example, if the rendering of the 3D environment only utilizes half of the display area, and the other half of the display area is used for other types of renderings, it may be difficult for viewers to identify certain types of activities in the display of the 3D environment. Specifically, it may be difficult for viewers to identify people who are speaking.

[0003] These issues can lead to many inefficiencies because it may require the user to carefully scan the user interface for relevant activities. This can be difficult when the user is viewing a rendering of a 3D environment with a large number of displayed avatars or a rendering that only utilizes a small portion of the display screen. Furthermore, when rendering a 3D environment, navigation tools are not always optimized to allow the user to search for relevant activities.

[0004] These shortcomings may result in ineffective interactions between computing devices and users, particularly during communication sessions. In addition, the above shortcomings of existing systems may result in a loss of user engagement. Computing devices that do not promote user engagement, or worse, promote a loss of user engagement and below-average interactions, may result in production losses and inefficiencies with respect to multiple computing resources. For example, when a user becomes fatigued or disengaged, when shared content is missed or ignored, the user may need to refer to other resources, such as documents, or use other forms of communication, such as text or chat messages. When a viewer misses a significant point or clue during a live conference, the missing content may need to be resent. Such activities may result in inefficient or duplicate use of networks, processors, memory, or other computing resources. Therefore, there is a constant need to develop improvements to help make the user experience of a communication system more consistent with or better than in-person meetings. Summary of the invention

[0005] The technology disclosed herein provides a system for automatically generating visual indicators to display the active speaker of a communication session displayed as a 3D representation. Some participants of a communication session can be displayed in a user interface using 3D representations (e.g., avatars) each located within a 3D environment. The user interface can also include multiple renderings of 2D images of other participants displayed in a gallery, such as a grid or other pattern of 2D images. When the operation refers to the display of "gallery mode", the system displays a user interface that includes multiple renderings of 2D presentations of 2D images of participants in a gallery arrangement (such as a grid or other pattern). The rendering in 2D format includes the rendering of a 2D image file, which may include a still image or a live stream of a 2D image. The user interface displayed in gallery mode may include a video stream or image of a participant from a 2D device (such as a personal computer (PC) with a 2D display screen and a single camera pointing to the user). For the purpose of illustration, the gallery is a display area designated for displaying a video or image of an active speaker. When a user displayed as a 3D representation begins speaking, the system may detect "speaker" activity via detecting an audio signal from the user device. Speaker activity may also be detected in other ways. For example, in some configurations, the speaker may be detected by analyzing the user's avatar. If the avatar movement indicates that the speaker is speaking, the system may then automatically add a supplemental image of the user to a gallery in the user interface. A gallery is a display area designated for active speakers. Such addition of a supplemental image of the user may include a live stream video of the user, an animation of the user, or another avatar of the user. The supplemental image may be displayed simultaneously with the 3D representation of the user located within the 3D environment.

[0006] The present invention provides many technical benefits. For example, by providing a system that automates visual indicators to display active speakers, particularly in communication systems, users can more efficiently exchange information and gain a better understanding of the broader context of the meeting. Moreover, by viewing the actual video stream of the active speaker, rather than relying on a displayed avatar with a reduced size, users can easily benefit from viewing subtle gestures and body language. This helps to mitigate the occurrence of missing or ignoring shared content. This can reduce the occurrence of users needing to resend information. More efficient communication of shared content can also help avoid the need for external systems (such as mobile phones for text messaging and other messaging platforms). This can help reduce the reuse of network, processor, memory or other computing resources, especially when extended meetings or additional meetings can be avoided.

[0007] Features and technical benefits in addition to those explicitly described above will be apparent by reading the following detailed description and reviewing the associated drawings. This summary is provided to introduce a selection of concepts in simplified form, which will be further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. For example, the term "technology" may refer to systems, methods, computer-readable instructions, modules, algorithms, hardware logic, and / or operations allowed by the above context and the entire document. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The detailed description is described with reference to the accompanying drawings. In the drawings, the left-most digit of a reference number identifies the drawing in which the reference number first appears. The same reference numbers in different drawings represent similar or identical items. References to individual items of a plurality of items may use reference numerals with letters of the alphabetical sequence to refer to each individual item. Generic references to items may use specific reference numerals without the alphabetical sequence.

[0009] Figure 1A A first user interface arrangement displaying a plurality of users is shown, wherein a first group of users is displayed as 3D representations and a second group of users is displayed as 2D images positioned in a designated area reserved for active speakers.

[0010] Figure 1B A second user interface arrangement is shown that provides a visual indicator to bring user focus to an active speaker of a first group of users displayed as 3D representations, wherein the visual indicator is a supplemental image of the active speaker displayed in a designated area within a 2D image grid.

[0011] Figure 2A A first user interface arrangement displaying a plurality of users is shown, wherein a first group of users is displayed as 3D representations and a second group of users is displayed as 2D images positioned in a designated area reserved for active speakers.

[0012] Figure 2B A second user interface arrangement is shown that provides a visual indicator to bring user focus to an active speaker of a first group of users displayed as a 3D rendering, wherein the visual indicator is an image of the active speaker displayed in a designated area located in an overflow area of ​​the user interface.

[0013] Figure 3A A first user interface arrangement showing a plurality of users is shown, wherein a first group of users are displayed as 3D representations and a second group of users are displayed as 2D images positioned as a designated area reserved for an active speaker.

[0014] Figure 3B A second user interface arrangement is shown that provides a visual indicator to bring user focus to an active speaker of a first group of users displayed as a 3D rendering, wherein the visual indicator is an image of the active speaker displayed in a designated area that is at least partially arranged to overlap a display of the 3D environment.

[0015] Figure 4A A scenario is shown in which a size threshold of a rendering of a 3D environment can be used to change the permission for allowing display of a visual indicator of an active speaker.

[0016] Figure 4B A scenario is shown in which a size threshold of a rendering of a 3D environment may be used to alter the display of a visual indicator to reject an active speaker.

[0017] Figure 5 Technical details of 2D and 3D rendering disclosed in this article are shown.

[0018] Fig. 6A A scene is shown in which a user representation is positioned within a 3D environment relative to a virtual camera.

[0019] Figure 6B Shown is how the system moves the virtual camera to a new position and orientation in response to detecting an active speaker.

[0020] Figure 7 is a flow chart illustrating aspects of a routine for controlling the display of visual indicators to show active speakers of a communication session displayed as a 3D representation.

[0021] Figure 8 is a computer architecture diagram showing an illustrative computer hardware and software architecture for a computing system capable of implementing the techniques and aspects of the techniques presented herein.

[0022] Fig. 9 is a computer architecture diagram illustrating a computing device architecture of a computing device capable of implementing the techniques and aspects of the techniques presented herein. DETAILED DESCRIPTION

[0023] Figure 1A and Figure 1BAn example of a UI transition is shown that changes the viewing perspective of a 3D environment to bring focus to relevant activities of users participating in a communication session.A communication session may be managed by a system 100 comprising a plurality of computers 11 , each computer corresponding to a plurality of users 10 . In this example, a first user 10A, Reta Taylor, is associated with a first computer 11A, a second user 10B, Miguel Silva, is associated with a second computer 11B, a third user 10C, Bryan Wright, is associated with a third computer 11C, a fourth user 10D, MJ Price, is associated with a fourth computer 11D, a fifth user 10E, Bruno Zhaos, is associated with a fifth computer 11E, a sixth user 10F, Semena Davis, is associated with a sixth computer 11F, a seventh user 10G, Krystal McKinney, is associated with a seventh computer 11G, an eighth user 10H, Jessica Kline, is associated with an eighth computer 11H, a ninth user 10I, Kat Larsson, is associated with a ninth computer 11I, and a tenth user 10J, Tracki Isaac, is associated with a tenth computer 11J. The users may also be referred to as "User A", "User B", etc. Other users (e.g., User K and User L) are also participants in the communication session.

[0024] Each user may be displayed in the user interface as a two-dimensional (2D) image 151, or each user may be displayed in the user interface as a three-dimensional (3D) representation 251. The 2D image 151 may be a static image or a video stream generated by a camera pointed at the user. The 3D representation 251 may be a static model or a dynamic model that animates in real-time action in response to user input, such as a user speaking or a user typing into a keyboard. The computer may be in the form of a desktop computer, a head mounted display unit, a tablet computer, a mobile phone, etc.

[0025] The system can generate a user interface showing various aspects of the communication session to each user participating in the communication session. In this example, a first user interface arrangement 101A is initially displayed to one or more users. The first user interface arrangement includes multiple renderings of each user 10 in two areas: a first area 130 designated for renderings of 3D representations of users positioned within the 3D environment 200 and a second area 131 designated for active speakers. In this embodiment, the second area 131 designated for active speakers is associated with a grid of 2D renderings of images that can be static images or dynamic video streams.

[0026] The system may select individual user renderings for the second area 131 that is actively speaking within the communication session. The system may use one or more thresholds or criteria to select individual users to be displayed within the second area 131 designated for active speakers. For example, when the system detects that a user is producing an audio signal from a microphone, the system may display a rendering of that user within the second area 131. In another example, the system may detect keyboard input or other forms of communication that may control the rendering of an avatar. If an input, such as a keyboard input for chatting, controls the avatar to give the appearance that the avatar is speaking, the system may display a second rendering of that user within the second area 131. The system may select a user using criteria such as speech rate, a threshold number of words or characters, or a threshold volume level to determine whether the input qualifies as a triggering input to invoke a UI transition described herein.

[0027] For example, if a user provides multiple keyboard or touch screen inputs to control an avatar or provide text that can be converted to speech, and the keyboard input includes a threshold number of words or a threshold speech rate, the system can display an image or a supplemental image of the user in the second area 131. The system can also analyze the audio signal to determine the speech rate, or determine whether the user has provided a threshold number of words in the speech input. If the speech or the number of words spoken by the user meets one or more criteria or exceeds one or more thresholds, the system can display a rendering of the user in the second area 131. Users with speech activity that do not meet one or more criteria or do not exceed one or more thresholds do not have their images displayed in the second area 131.

[0028] The second area 131 may also have a predetermined limit of users that may be displayed. In such an embodiment, users participating in voice activity are ranked according to their activity. For example, users speaking in a communication session may be ranked in a speaker queue based on speech rate, volume, use of selected words or terms, etc. The highest ranked speakers, up to a predetermined limit, may be displayed in the second area 131. Once the input of those displayed speakers does not meet the criteria, the image of the active speaker may be removed.

[0029] The first user interface arrangement 101A may include two-dimensional (2D) images 151 of some users, which may include still images or live video feeds of the users. The first user interface arrangement 101A may also include renderings of 3D representations 251, which may include avatars positioned within the 3D virtual environment 200. In this particular example, the first user interface arrangement 101A includes renderings of a 3D representation 151A of a first user 10A, a rendering of a 3D representation 251B of a second user 10B, and a rendering of a 3D representation 251C of a third user 10C.

[0030] The first user interface arrangement 101A also includes a rendering of a 2D image 151D of a fourth user 10D, a rendering of a 2D image 151E of a fifth user 10E, a rendering of a 2D image 151F of a sixth user 10F, a rendering of a 2D image 151G of a seventh user 10G, and a rendering of a 2D image 151H of an eighth user 10H. For illustrative purposes, the first user interface arrangement 101A is displayed on an eighth computer of the eighth user 10H. Therefore, the rendering of the 2D image 151H of the eighth user 10H is a display of a local user, such as a "me" video.

[0031] While displaying the first user interface arrangement 101A, the system monitors the activities of the users displayed within the 3D environment. In order to preserve computing resources, in some configurations, the system may limit monitoring activities to the audio streams of the users displayed as 3D representations 251 in the 3D environment. This allows the system to run sound analysis only on certain users' streams. Figure 1A In the example of , it would include audio streams of a first user, a second user, and a third user. As described herein, the system can determine whether one of the audio streams meets one of our criteria. For example, if the first user 10A represented by the 3D representation 251A begins speaking at a threshold rate and / or a threshold volume, the system can determine that the first user is the active speaker of the user having the 3D representation displayed in the 3D environment.

[0032] In response to determining that a user having a 3D representation displayed in a 3D environment has voice input that satisfies one or more criteria, the system performs the following steps: Figure 1A The first user interface arrangement 101A to Figure 1B As shown, in the transition, the system adds a second rendering of the image or representation of the active speaker in the second user interface arrangement 101B.

[0033] In some embodiments, the second image of the active speaker is displayed in the second area 131 reserved for the active speaker. Continuing with the above example, when the first user 10A, displayed as the first 3D representation 251A, provides a speech input that satisfies one or more criteria, such as Figure 1B , the system displays a second rendering of a 2D image 151A of a first user 10A. In this example, the 2D image of the first user includes a video stream generated by a camera of a first computing device associated with the first user 10A. The 2D image 151A of the first user 10A is located within a second area 131 reserved for active speakers. In this embodiment, the second area 131 reserved for active speakers is in a grid format, where each portion of the grid shows a video rendering of a participant. The area can be limited to a predetermined number of participants as described herein.

[0034] The second user interface arrangement 101B is also configured with a 2D image 151A of the first user 10A so that it is displayed simultaneously with the rendering of the 3D representation 251A of the first user 10A. In some embodiments, the rendering of the 3D representation 251A of the first user 10A can be maintained so that the first display area 130 displays the same viewing angle of the 3D environment 200 throughout the transition.

[0035] A second rendering of an image or representation of an active speaker, also referred to herein as a "supplemental image" or "second image," may be removed based on one or more factors. In some embodiments, the supplemental image may be displayed for a predetermined period of time. After a predetermined period of time, the supplemental image, such as the 2D image 151A of the first user 10A, may be removed and replaced with another active speaker in the speaker queue. In some embodiments, the supplemental image may be displayed until one or more criteria for the voice input are no longer met. For example, if one or more criteria for the voice input include a speech rate, and a second image of the user providing the voice input is displayed in the second area 131 in response to voice input that meets or exceeds the speech rate, the system may remove the second image of the user once the user stops speaking for a predetermined period of time.

[0036] In another example, if a second image of the user providing the voice input is displayed in the second area 13 in response to the voice input satisfying one or more criteria (eg, Figure 1B In one embodiment, the system may remove the second image of the user once the user stops speaking or the user's voice input no longer meets one or more criteria. In other embodiments, the system may remove the second image of the user once the user stops speaking for a predetermined period of time. The system may remove the second image of the user based on other factors. For example, if other users speak at a louder volume than the user displayed in the second area, the images of those other users may replace the user's second image. These examples are provided for illustrative purposes and should not be construed as limiting. It will be appreciated that any type of predetermined input metric associated with any number of words, number of sentences, number of phrases, or any input rate may be used to determine when to display and / or remove the user's second image.

[0037] Figure 2A-2B Another example of a user interface transition that can occur when a triggering input is received is shown. For illustrative purposes, the triggering input may include any type of voice input that meets one or more criteria described herein. The triggering input may include any type of user input that causes the 3D representation to perform a gesture that gives the appearance that the 3D representation is speaking. The triggering input may include any type of user input that transmits any content to another user, including but not limited to a text message, an instant message, a group chat message, etc.

[0038] This example is similar to the above example in that the first user 10A provides voice input or another qualified input, and in response to the input, a second image of the user is displayed within the second area 131 reserved for active speakers. Figure 2A In the example of , a second area 131 reserved for active speakers is located in association with an overflow queue. The overflow queue can be an area of ​​the UI that displays the rendering of users that are not qualified (e.g., ranked high enough) to be displayed in the 2D rendered grid. When the speaker queue for the grid rendering has reached a maximum limit of users, an overflow queue, also referred to herein as a supplemental queue, can be generated. For example, if the speaker queue of the grid portion of the UI has a limit of three users, and there are four active speakers, the system will display the first three users in the image grid and display the fourth ranked speaker in the overflow queue.

[0039] like Figure 2A As shown, the first user interface arrangement 201A shows a rendering of a 3D representation of a first user 10A displayed within a rendering of a 3D environment. When the first user 10A begins to provide voice input or any other qualifying input, for example, input that controls the 3D representation to give the appearance that the user is speaking, the system then displays a supplemental image 151A of the user within the second area 131, as shown in FIG. Figure 2B shown.

[0040] In some configurations, an overflow queue may be reserved for users participating in a communication session with a computer that does not produce a video component. Thus, a user participating as an audio-only user may be placed in a queue that is not a video component. Figure 2A and 2B The overflow queue is shown in . Therefore, Figure 2A The second area shown in Figure 1A The second region shown in FIG. Figure 1A The second area shown in can be reserved for an active speaker who is using a communication device that can generate a video stream and an audio stream. The video stream can also include a rendering of a 3D avatar. Figure 1A The second area shown in can be reserved for active speakers who are using a communication device that is generating an audio stream but not a video stream. The user interface can also include graphical elements, such as a ring around audio-only users, which can depict users in an overflow queue (e.g., Figure 2A The second area 131 shown in FIG. 1 changes color, shape or size when the user speaks. The system can monitor the audio signal of each audio-only user and modify the graphical element when a threshold volume is detected.

[0041] Figure 3A-3BAnother example of a user interface transition that may occur when a trigger input is received is shown. This example is similar to the above examples in that a first user 10A provides a trigger input, and in response to the trigger input, a second image of the user is displayed within a second area 131 reserved for active speakers. In this embodiment, the second area 131 reserved for active speakers is positioned and arranged to at least partially overlap with a 3D rendering of a 3D environment 200. In other words, in this embodiment, the first area 130 and the second area 131 reserved for rendering of a 3D representation of a user positioned within the 3D environment 200 are arranged to at least partially overlap.

[0042] like Figure 3A As shown, the first user interface arrangement 301A shows a rendering of a 3D representation of a first user 10A displayed within a rendering of a 3D environment. When the first user 10A begins to provide voice input or any other input that controls the 3D representation to give the appearance that the user is speaking, such as Figure 3B As shown, the system then displays the second UI arrangement 301B, ie, displays the supplementary image 151A of the user in the second area 131 .

[0043] Figure 4A and 4B Aspects of a system that can control the user interface transitions described herein are shown. In some configurations, the user interface transitions can be controlled by one of our factors, such as the size of the 3D rendering of the 3D environment 200. This control can be utilized to preserve computing resources in situations where a transition is more likely to be needed. For example, if the 3D rendering of the 3D environment 200 is relatively small within the user interface, the system can cause the transitions described herein to help the user identify the active speaker within the 3D environment. In scenarios where the rendering of the 3D environment is relatively small and it is difficult for the user to see the actual movement of the 3D avatar 251, the display of a notification of the active speaker may be necessary.

[0044] In some embodiments, if the rendering of 3D environment 200 has one or more dimensions that are less than a threshold dimension, the system can set permissions to allow the transformations described herein. Figure 4A , where at least one dimension of the rendering of the 3D environment 200 is less than a threshold size or threshold dimension. Figure 4B As shown in the example of , at least one dimension of the rendering of the 3D environment 200 is greater than a threshold size or a threshold dimension. When this condition is detected, the system can set permissions for denying the UI transitions described herein, for example, the system can limit the display of a graphical notification of an active speaker with a 3D representation located in the 3D environment. This allows the system to conserve resources by limiting user interface transitions (e.g., display of notifications of an active speaker) in scenarios where the transition is unlikely to be needed.

[0045] Figure 5 3 shows additional technical details of UI rendering disclosed herein. In some embodiments, upon receiving input for causing a UI transition described herein, the system can determine a viewing angle for controlling the position and orientation of the virtual camera 350 for the 3D representation of the active speaker. For example, in Figure 1A In the example shown, when a first user 10, displayed as a 3D representation within a 3D environment, provides a trigger input (such as a voice input that satisfies one or more criteria), the system can modify the position or orientation of the virtual camera 350 so that the rendering of the 3D environment provides a more direct view of the user's face.

[0046] For example, Fig. 6A As shown, before the trigger input, consider a scenario where the position and orientation of the virtual camera 350 is configured such that the 3D representation 251A of the first user 10A is facing away from the virtual camera. In this case, a viewer of the 3D representation 251A of the first user 10A may not be able to see when the user is speaking because they will not be able to see the facial gestures based on the camera angle. Therefore, in response to the trigger input, in addition to providing the supplemental image 151A, as shown in FIG. Figure 6B As shown, the system can also modify the position and orientation of the virtual camera so that the rendering of the 3D environment also shows the face of the 3D representation of the first user. This modification to the orientation and / or position of the virtual camera can be modified so that a viewer of the 3D environment 200 will be able to see the face of the active speaker. This modification to the virtual camera can provide further notification to a viewer of the active speaker. In addition, movement of the orientation or position of the virtual camera can be initiated in response to any trigger input described herein.

[0047] Now return to Figure 5 , additional aspects of a system configured to implement the technology disclosed herein are shown and described. For illustrative purposes, a rendering of a 2D image file or a rendering of a 2D image of a user may be generated by receiving 2D image data 310 (e.g., an image file) via a 2D rendering engine 551. The rendering of a 2D image file may include a 2D environment (e.g., the background of an image) and a 2D object (e.g., an image of a person or an avatar). An image file (e.g., image data 310) may have pixels arranged in two dimensions, for example, pixels arranged in a two-dimensional coordinate system (x, y). The data may also be referred to herein as a two-dimensional model based on a two-dimensional coordinate system. Each portion of the image may be a pixel or any other geometric shape, such as a triangle. For example, a group of pixels or triangles may be used to generate a rendering of a two-dimensional avatar of a user or a live video image of a person.

[0048] A two-dimensional environment with multiple 2D images of participants of a communication session is also referred to herein as a "grid environment". The image data or the communication data stream may define a two-dimensional environment or a two-dimensional object, and the two-dimensional environment may be rendered on a display screen. The rendering may be referred to herein as a two-dimensional rendering of a two-dimensional environment or a two-dimensional rendering of a two-dimensional object. This is also referred to herein as "rendering of a two-dimensional image".

[0049] For illustrative purposes, a rendering of a 3D model or a rendering of a 3D representation of a user can be generated by a 3D rendering engine 552 that accesses 3D model data 320 (e.g., a 3D model). The 3D model may include parameters that define the 3D environment 200, such as a model of a room, and parameters that define 3D objects, such as the size, shape, and position data of a representation 351 of a user or other virtual object. A three-dimensional environment is a computational environment model based on a three-dimensional coordinate system. The properties of a three-dimensional environment and a three-dimensional object in the three-dimensional environment are based on components positioned within a three-dimensional coordinate system (x, y, z). Each component can be a triangle or any other geometric shape. Each component can have a position, such as a position in a three-dimensional coordinate system, and an orientation, such as the direction in which the triangle points. For example, a set of triangles can be used to generate a rendering of a three-dimensional avatar of a user or a three-dimensional rendering of a three-dimensional object.

[0050] The three-dimensional environment is also referred to as an "immersive environment" in this article. Model data or a three-dimensional model can be included in the communication data stream, and the model data can define the three-dimensional environment. The three-dimensional environment can be based on a three-dimensional coordinate system. When the rendering engine 552 generates a 3D rendering from the 3D model, the rendering is generated from a reference point in the environment (e.g., a perspective with a position relative to the virtual environment). For illustrative purposes, the reference point is also referred to as a virtual camera 350 in this article. The camera can have a field of view for generating a rendering of a 3D environment or a 3D object based on the position of the virtual camera 350. The rendering of a three-dimensional object in a three-dimensional environment is based on the position and orientation of the three-dimensional object and the position of the virtual camera 350.

[0051] In some embodiments, a two-dimensional image can be displayed within a three-dimensional environment. For example, this can occur when a communication system receives a two-dimensional video stream of a user, but the participant receiving the video stream is viewing a 3D environment using an HMD. This can cause the system to display the image of the user as if they were appearing on a virtual television on the wall of the virtual environment. This is referred to herein as a two-dimensional rendering of the user within a three-dimensional environment. This can include Figure 2A The third user 10C is shown in FIG. 1 as rendering 151C.

[0052] In some embodiments, the three-dimensional environment and three-dimensional objects defined by the three-dimensional model can be displayed as two-dimensional renderings. For example, when a communication session involves a user interface that displays a two-dimensional image, such as when Teams is in grid mode, this can occur. In this mode, the system may need to display images of users interacting in a 3D environment. In this case, a 2D image of the 3D environment is displayed from a specific position (e.g., a virtual camera position), and a 2D image is displayed within a grid in the grid. This rendering may be referred to as a two-dimensional rendering of a three-dimensional environment in this article. In order to achieve two-dimensional rendering of a three-dimensional environment, a transformation can be used to project the model data defining the three-dimensional environment. The transformation can generate a rendering so that the width, height, and depth of a three-dimensional object can be expressed on a flat screen using a vector projection from the model of the object to the viewpoint (e.g., a virtual camera position).

[0053] Figure 7 is a diagram illustrating aspects of a routine 700 for providing an automation of visual indicators, showing active speakers of 3D representations of users participating in a communication session. It should be understood by those of ordinary skill in the art that the operations of the methods disclosed herein are not necessarily presented in any particular order, and it is contemplated that some or all operations may be performed in an alternative order. For ease of description and illustration, the operations are presented in the order of demonstration. Operations may be added, omitted, performed, and / or performed simultaneously without departing from the scope of the appended claims.

[0054] It should also be understood that the method shown can be started or ended at any time and does not need to be performed as a whole. Some or all of the operations of the method and / or substantially equivalent operations can be performed by executing computer-readable instructions included on a computer storage medium, as defined herein. As used in the specification and claims, the term "computer-readable instructions" and its variants are widely used herein to include routines, applications, application modules, program modules, programs, components, data structures, algorithms, etc. Computer-readable instructions can be implemented on various system configurations, including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based programmable consumer electronics, combinations thereof, etc. Although the example routines described below operate on a system (e.g., one or more computing devices), it is understood that the routines can be executed on any computing system, which can include any number of computers working together to perform the operations disclosed herein.

[0055] Therefore, it should be understood that the logical operations described herein are implemented as a sequence of computer-implemented actions or program modules running on a computing system such as described herein and / or as interconnected machine logic circuits or circuit modules within the computing system. The implementation is a matter of choice depending on the performance and other requirements of the computing system. Therefore, the logical operations may be implemented in software, firmware, dedicated digital logic, and any combination thereof.

[0056] in addition, Figure 7 The operations shown in the and other figures can be implemented in association with the example user interfaces and systems described herein. For example, the various devices and / or modules described herein can generate, send, receive, and / or display data associated with the content of a communication session (e.g., live content, broadcasted events, recorded content, etc.) and / or a rendering UI including a rendering of one or more participants of a remote computing device, avatar, channel, chat session, video stream, image, virtual object, and / or application associated with the communication session.

[0057] Routine 700 includes operation 703, in which the system causes display of a first user interface arrangement having a 3D representation. An example of a first user interface arrangement is shown in FIG. Figure 1A . The first user interface arrangement may include a rendering of a 3D environment 200 having a plurality of three-dimensional representations of users participating in a communication session. Each of the three-dimensional representations has an independent position and orientation within the three-dimensional environment, and each three-dimensional representation is controlled by input provided by an associated user of the plurality of users. The first user interface arrangement may also include regions each reserved for a different type of rendering. For example, a first region is reserved for the 3D environment, and a second region is reserved for a rendering of an active speaker.

[0058] At operation 705, the system receives input identifying an active speaker. The operation may include using one or more modules to monitor audio streams received from computing devices of users participating in a communication session. When one of the audio streams includes the user's voice and the voice meets one or more criteria, the system may initiate one or more actions to convert the user interface arrangement.

[0059] Operation 705 may also include determining that the user (10A) is a member (251A) of the users (10) rendered in the 3D representation (251). The system may determine that the user is a member of the users rendered as 3D representations via a data structure that associates user identities with individuals depicted in the 3D environment 200. Any conference participant identified as a user displayed as a 3D representation in the 3D environment is a member of the users identified in the data structure. Thus, the system may update the data structure each time a user is rendered as a 3D representation in the 3D environment. Furthermore, when a user is no longer displayed as a 3D representation in the 3D environment, the system may update the data structure to remove the user identity. In some embodiments, then in response to the system determining that a user such as the first user (10A) is identified as a member (251) of the users rendered as a 3D representation, the system may perform any of the disclosed operations including operation 707. Thus, in such embodiments, if the system determines that the user is not a member (251) of the users rendered as a 3D representation, the system may limit the performance of operations, including operation 707.

[0060] At operation 707, the system generates a second rendering of the active speaker in response to the input satisfying the one or more criteria. The generation of the second rendering may include Figure 1A The first user interface arrangement shown is converted to Figure 1B A similar conversion may be performed, such as from Figure 2A The first user interface shown in FIG. Figure 2B , and from Figure 3A The first user interface shown in FIG. Figure 3B A transition to the second user interface arrangement shown in .

[0061] In operation 707, the system may control the user interface transition based on the size of the rendering of the 3D environment. For example, if the rendering of the 3D environment is less than a threshold size, the system may allow one or more computing devices to perform the user interface transition described herein. When the rendering of the 3D environment is greater than the threshold size, the system may restrict one or more computing devices from performing the user interface transition described herein.

[0062] In another embodiment, if the rendering of the 3D environment occupies less than a predetermined percentage of the display area of ​​the device (e.g., the device screen), the system may allow one or more computing devices to perform the user interface transitions described herein. When the rendering of the 3D environment occupies more than a threshold percentage of the display area of ​​the device, the system may restrict one or more computing devices from performing the user interface transitions described herein.

[0063] In some embodiments, if the title or role of the user does not meet one or more criteria, the system blocks the conversion of the first user interface arrangement to the second user interface arrangement. In addition, if the title or role of the user meets one or more criteria, the system allows the conversion of the first user interface arrangement to the second user interface arrangement. For example, if the level of a person in an organization, the performance level of a person, or any other status information of a person meets one or more thresholds, the system may allow one or more computing devices to perform the user interface conversion described herein. If the level of a person in an organization, the performance level of a person, or any other status information of a person does not meet one or more thresholds (such as performance thresholds, ranking thresholds, etc.), the system may restrict one or more computing devices from performing the user interface conversion described herein. For example, the system may allow a UI conversion for a CEO, for example, generating a notification of an active 3D speaker, but may not allow a UI conversion for a person at another level. In another example, the system may provide a notification of a view of an active speaker who has met the sales quantity in the organization (e.g., performing a UI conversion), but the system may not provide a notification for an active speaker for a person who has not yet met the sales quantity in the organization.

[0064] In operation 707, in some embodiments, the second user interface arrangement includes a first area reserved for rendering of the three-dimensional environment and a second area reserved for rendering of the active speaker of the communication session. The second area is a grid format with individual portions for each user rendering, such as Figure 1A The individual renderings 151 are shown. The transition of the first user interface arrangement to the second user interface arrangement includes adding a new grid section for the second rendering of the user. Thus, all other active speakers of the second region depicted before the transition will remain in the user interface throughout the transition.

[0065] In some embodiments, the generation of the second rendering may include generating a new user interface, such as a new window, that is independently controlled by the user. This enables the user to view the second presentation of the active speaker in a window that can be resized, moved, or closed in response to input from the viewing user. The window can be in any position and size relative to the rendering of the 3D environment.

[0066] In operation 709, the system may also modify the perspective of the 3D representation of the active speaker to bring the focus to the user. Fig. 6A and Figure 6B As shown, the virtual camera 350 used to determine the viewing angle of the 3D environment may not always be pointed at the face of the avatar. When the user provides input that causes the avatar to perform a voice gesture, the system can change the camera position and orientation to allow the virtual camera to capture the avatar's voice gesture. In addition to generating a supplemental rendering of the user in the user interface transition, this modification of the virtual camera angle can also bring focus to the user's activities.

[0067] In response to determining that the input no longer meets one or more criteria, the system can remove the second rendering of the user interface. For example, if the user has not spoken for a predetermined period of time, the system can remove the 2D image of the user and return to displaying only the 3D rendering of the user. Removal of the second rendering can also occur when the position of the person within the queue or ranking drops below one or more thresholds, for example, another user becomes ranked higher, etc.

[0068] In some environments, the system may also trigger a transition of the user interface based on other signals besides detecting sound input. For example, if the system determines that the rendering of the 3D representation of a user is too small, e.g., the 3D rendering is less than a threshold size, the system may automatically display a second 2D rendering of the user. In some configurations, if the person is speaking and their 3D rendering is also less than the threshold size, the system may also display a second additional 2D rendering of the user. Thus, the size of the 3D rendering may be used as a filter for a predetermined user who is speaking. Thus, if a person has a 3D representation greater than a threshold size and the user's voice meets one or more criteria, the user may not be displayed in the second 2D rendering. However, if the person has a 3D representation less than a threshold size and the user's voice meets one or more criteria, the user may be displayed in the second 2D rendering.

[0069] Additionally, after displaying the user as a second additional 2D image in response to detecting that the user's voice speech input meets one of our criteria, the system can also increase the 2D image if the user speaks for more than a predetermined period of time or if the user's presentation contains more than a threshold number of words, phrases, etc. Thus, after the user interface transitions, in response to the user's voice input meeting a second set of criteria (e.g., the user speaks for longer than a second threshold time or uses more than a second threshold number of words or phrases), the user's 2D image can be increased to full screen or at least magnified.

[0070] The following clauses further describe the present disclosure.

[0071] Item A: A method for generating a visual indicator for distinguishing an active speaker from a user (10) of a communication session (604) displayed as a 3D representation (251), the method being configured for execution on a system, the method comprising: causing a first user interface arrangement (101A) to be displayed that includes individual renderings of three-dimensional representations (251) of a plurality of users (10) participating in the communication session, wherein each of the three-dimensional representations (251) has an independent position and orientation within the three-dimensional environment (200), the independent position and orientation being independent of each other. The first user interface arrangement (101A) further comprises a rendering of a subset of users (151D-151F) in 2D format, the rendering of the subset of users being generated by a 2D device (11D-11F) associated with the subset of users (151D-151F); receiving input from the plurality of users identifying a user (10A) as the active speaker, wherein the user (10A) is identified by detecting voice input received by a microphone associated with the user The invention relates to a method for providing a user interface for displaying a plurality of three-dimensional representations of the user (10) rendered in a 3D manner and for displaying the plurality of three-dimensional representations of the user (10) rendered in a 3D manner; and a method for displaying a plurality of three-dimensional representations of the user (10) rendered in a 3D manner and for displaying the plurality of three-dimensional representations of the user (10) rendered in a 3D manner. 51) and a second additional rendering (151A) in 2D format of the user (10A), wherein: the first user interface arrangement (101A) does not simultaneously display the 3D representation (251A) of the user (10A) and the second additional rendering (151A) in 2D format of the user (10A), and the second user interface arrangement (101B) simultaneously displays the second additional rendering (151A) of the user (10A) and the 3D representation (251A) of the user (10A) located within the three-dimensional environment (200).

[0072] Item B: A method according to any one of Item B: wherein the second user interface arrangement includes a first area reserved for rendering of the three-dimensional environment and a second area reserved for rendering of an active speaker of the communication session, the second area including a 2D rendering of a video stream of a user who qualifies as an active speaker, wherein the second rendering of the user (10A) is at least partially displayed within the second area.

[0073] Item C: A method according to any of the items, wherein the second user interface arrangement comprises: a first area, the first area being reserved for rendering of the three-dimensional environment; and a second area, the second area being reserved for rendering of the active speaker, the active speaker being eligible to enter an overflow queue of users secondary to a main queue of users, wherein the second rendering of the user (10A) is at least partially displayed within the second area.

[0074] Clause D: A method according to any of the clauses, wherein the second user interface arrangement includes a first area reserved for rendering of the three-dimensional environment and a second area reserved for rendering of an active speaker of the communication session, the second area at least partially overlapping the first area, wherein the second rendering of the user (10A) is at least partially displayed within the second area.

[0075] Item E: A method according to any one of the items, wherein the system controls the conversion of the first user interface arrangement to the second user interface arrangement based on the size of the rendering of the three-dimensional environment, wherein when the size of the rendering of the three-dimensional environment is less than a size threshold, the system prevents the conversion of the first user interface arrangement to the second user interface arrangement, and wherein when the size of the rendering of the three-dimensional environment is greater than the size threshold, the system allows the conversion of the first user interface arrangement to the second user interface arrangement.

[0076] Clause F: A method according to any of the clauses, wherein the system controls the transition of the first user interface arrangement to the second user interface arrangement based on the title or role of the user, wherein if the title or the role of the user does not meet one or more criteria, the system prevents the transition of the first user interface arrangement to the second user interface arrangement, wherein if the title or the role of the user meets one or more criteria, the system allows the transition of the first user interface arrangement to the second user interface arrangement.

[0077] Clause G: According to a method described in any of the clauses, the second user interface arrangement includes a first area reserved for rendering of a three-dimensional environment and a second area reserved for rendering of an active speaker of a communication session, wherein the second area is in a grid format having individual portions for each user rendering, and wherein conversion of the first user interface arrangement to the second user interface arrangement includes adding a new grid portion for the second rendering of the user.

[0078] Figure 8600 is a diagram illustrating an example environment 600 in which a system 602 can implement the techniques disclosed herein. It should be understood that the above-described subject matter can be implemented as a computer-controlled device, a computer process, a computing system, or an article such as a computer-readable storage medium. The operations of the example methods are shown in individual blocks and are summarized with reference to these blocks. The method is shown as a logical flow of blocks, each of which can represent one or more operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, enable the one or more processors to perform the operations.

[0079] Typically, computer executable instructions include routines, programs, objects, modules, components, data structures, etc. that perform specific functions or implement specific abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be performed in any order, combined in any order, subdivided into multiple sub-operations, and / or performed in parallel to implement the described processes. The described processes may be performed by resources associated with one or more devices such as one or more internal or external CPUs or GPUs, and / or one or more hardware logic such as a field programmable gate array ("FPGA"), a digital signal processor ("DSP"), or other types of accelerators.

[0080] All of the methods and processes described above may be embodied in software code modules executed by one or more general-purpose computers or processors and fully automated. The code modules may be stored in any type of computer-readable storage medium or other computer storage device, such as those described below. Some or all of the methods may alternatively be embodied in dedicated computer hardware, such as the computer hardware described below.

[0081] Any routine description, element or block in the flowcharts described herein and / or depicted in the accompanying drawings should be understood to potentially represent a module, segment or portion of code, which includes one or more executable instructions for implementing a specific logical function or element in the routine. Alternative implementations are included within the scope of the examples described herein, where elements or functions may be deleted or performed from the order shown or discussed, including substantially synchronously or in reverse order, depending on the functionality as will be understood by those skilled in the art.

[0082] In some implementations, system 602 can be used to collect, analyze, and share data displayed to users of communication session 604. As shown, communication session 603 can be implemented between multiple client computing devices 606(1)-606(N) (where N is a number having two or greater values) that are associated with or part of system 602. Client computing devices 606(1)-606(N) enable users, also referred to as individuals, to participate in communication session 603.

[0083] In this example, communication session 603 is hosted by system 602 over one or more networks 608. That is, system 602 can provide services that enable users of client computing devices 606(1) to 606(N) to participate in communication session 603 (e.g., via live viewing and / or recorded viewing). Thus, "participants" of communication session 603 can include users and / or client computing devices (e.g., multiple users can be in a room participating in a communication session via the use of a single client computing device), each of which can communicate with other participants. Alternatively, communication session 603 can be hosted by one of client computing devices 606(1) to 606(N) using peer-to-peer technology. System 602 can also host chat conversations and other team collaboration functionality (e.g., as part of an application suite).

[0084] In some implementations, such chat conversations and other team collaboration functions are considered external communication sessions distinct from communication session 603. The computing system 602 that collects participant data in communication session 603 may be able to link to such external communication sessions. Thus, the system may receive information that enables connection to such external communication sessions, such as date, time, session details, etc. In one example, a chat conversation may be conducted in accordance with communication session 603. Additionally, system 602 may host communication session 603, which includes at least a plurality of participants located at a meeting location (such as a conference room or auditorium) or located at different locations.

[0085] In the example described herein, client computing devices 606(1) to 606(N) participating in a communication session 603 are configured to receive and render communication data for display on a user interface of a display screen. The communication data may include a collection of various instances or streams of live content and / or recorded content. The collection of various instances or streams of live content and / or recorded content may be provided by one or more cameras (such as a video camera). For example, an individual stream of live or recorded content may include media data associated with a video feed provided by a camera (e.g., audio and visual data capturing the appearance and voice of a user participating in the communication session). In some embodiments, the video feed may include such audio and visual data, one or more still images, and / or one or more avatars. The one or more still images may also include one or more avatars.

[0086] Another example of an individual stream of live or recorded content may include media data including avatars of users participating in a communication session and audio data capturing the user's voice. Yet another example of an individual stream of live or recorded content may include media data including a file displayed on a display screen and audio data capturing the user's voice. Thus, the various streams of live or recorded content within the communication data enable teleconferencing between a group of people and sharing content within a group of people. In some embodiments, the various streams of live or recorded content within the communication data may originate from multiple co-located cameras positioned in a space, such as a room, to record or stream a presentation including a presentation and one or more individuals consuming the presented content.

[0087] Participants or attendees can view the content of the communication session 603 in real time as the activity occurs or alternatively via a recording at a later time after the activity occurs. In the example described herein, the client computing devices 606(1) to 606(N) participating in the communication session 603 are configured to receive and render communication data for display on a user interface of a display screen. The communication data may include a collection of various instances or streams of live and / or recorded content. For example, an individual stream of content may include media data associated with a video feed (e.g., audio and visual data that captures the appearance and voice of users participating in the communication session). Another example of an individual stream of content may include media data that includes an avatar of a user participating in the conference session and audio data that captures the user's voice. Yet another example of an individual stream of content may include media data that includes content items displayed on a display screen and / or audio data that captures the user's voice. Thus, the various streams of content within the communication data enable facilitating a conference or broadcast presentation between a group of people dispersed in remote locations.

[0088] Participants or attendees of a communication session are people within range of a camera or other image and / or audio capture device so that the person's movements and / or sounds generated while the person is viewing and / or listening to content shared via the communication session can be captured (e.g., recorded). For example, a participant may be sitting in a crowd viewing the shared content in real time at a broadcast location where a stage presentation is occurring. Alternatively, a participant may be sitting in an office conference room viewing the shared content of a communication session with other colleagues via a display screen. Further, a participant may be sitting or standing in front of a personal device (e.g., a tablet computer, smart phone, computer, etc.) viewing the shared content of the communication session individually in their office or home.

[0089] Figure 8 System 602 includes device 610. Device 610 and / or other components of system 602 may include distributed computing resources that communicate with each other and / or with client computing devices 606(1)-606(N) via one or more networks 608. In some examples, system 602 may be a stand-alone system responsible for managing various aspects of one or more communication sessions, such as communication session 603. As an example, system 602 may be managed by an entity such as Slack, WEBEX, GOTOMEETING, GOOGLE HANGOUTS, etc.

[0090] The network 608 may include, for example, a public network such as the Internet, a private network such as an institutional and / or personal intranet, or some combination of private and public networks. The network 608 may also include any type of wired and / or wireless network, including but not limited to a local area network ("LAN"), a wide area network ("WAN"), a satellite network, a cable network, a Wi-Fi network, a WiMax network, a mobile communication network (e.g., 3G, 4G, etc.), or any combination thereof. The network 608 may utilize communication protocols, including packet-based and / or datagram-based protocols, such as the Internet Protocol ("IP"), the Transmission Control Protocol ("TCP"), the User Datagram Protocol ("UDP"), or other types of protocols. In addition, the network 608 may also include multiple devices that facilitate network communications and / or form the hardware foundation of the network, such as switches, routers, gateways, access points, firewalls, base stations, repeaters, backbone devices, etc.

[0091] In some examples, network 608 may also include devices capable of connecting to a wireless network, such as a wireless access point ("WAP"). Examples support connections through WAPs that send and receive data over various electromagnetic frequencies (e.g., radio frequencies), including WAPs that support the Institute of Electrical and Electronics Engineers ("IEEE") 802.11 standards (e.g., 802.11g, 802.11n, 802.11ac, etc.) as well as other standards.

[0092] In various examples, device 610 may include one or more computing devices that operate in a cluster or other grouping configuration to share resources, balance loads, increase performance, provide failover support or redundancy, or for other purposes. For example, device 610 may belong to various types of devices, such as traditional server-type devices, desktop computer-type devices, and / or mobile devices. Therefore, although shown as a single type of device or server-type device, device 610 may include a variety of device types and is not limited to a particular type of device. Device 610 may represent, but is not limited to, a server computer, a desktop computer, a web server computer, a personal computer, a mobile computer, a laptop computer, a tablet computer, or any other type of computing device.

[0093] The client computing device (e.g., one of client computing devices 606(1)-606(N)) (each client computing device also referred to herein as a "data processing system") can be of various types of devices, which can be the same as or different from device(s) 610, such as a traditional client-type device, a desktop-type device, a mobile-type device, a dedicated-type device, an embedded device, and / or a wearable-type device. Thus, client computing devices may include, but are not limited to, desktop computers, gaming consoles and / or gaming devices, tablet computers, personal data assistants (“PDAs”), mobile phone / tablet hybrid devices, laptop computers, telecommunication devices, computer navigation type client computing devices such as satellite-based navigation systems including global positioning system (“GPS”) devices, wearable devices, virtual reality (“VR”) devices, augmented reality (“AR”) devices, implanted computing devices, automotive computers, web-enabled televisions, thin clients, terminals, Internet of Things (“IoT”) devices, workstations, media players, personal video recorders (“PVRs”), set-top boxes, cameras, integrated components (e.g., peripherals) for inclusion in computing devices, appliances, or any other type of computing device. Furthermore, client computing devices may include combinations of the previously listed examples of client computing devices, such as, for example, desktop computer type devices or mobile type devices combined with wearable devices, etc.

[0094] The various classes and device types of client computing devices 606(1) through 606(N) may represent any type of computing device having one or more data processing units 692 that are operably connected to a computer-readable medium 694, such as via a bus 616, which in some cases may include a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and one or more of any of a variety of local, peripheral, and / or independent buses.

[0095] Executable instructions stored on computer-readable medium 694 may include, for example, operating system 619 , client module 620 , profile module 622 , and other modules, programs, or applications that may be loaded and executed by data processing unit(s) 692 .

[0096] The client computing devices 606(1)-606(N) may also include one or more interfaces 624 to enable communication between the client computing devices 606(1)-606(N) and other networked devices (such as device 610) via the network 608. Such network interfaces 624 may include one or more network interface controllers (NICs) or other types of transceiver devices to send and receive communications and / or data via the network. In addition, the client computing devices 606(1)-606(N) may include input / output ("I / O") interfaces (devices) 626 that enable communication with user input devices such as peripheral input devices (e.g., game controllers, keyboards, mice, pens, voice input devices such as microphones, video cameras for obtaining and providing video feeds and / or still images, touch input devices, gesture input devices, etc.) and / or output devices such as peripheral output devices (e.g., displays, printers, audio speakers, tactile output devices, etc.). Figure 8 Client computing device 606 ( 1 ) is shown connected in some manner to a display device (eg, display screen 629 (N)) that can display a UI in accordance with the techniques described herein.

[0097] exist Figure 8 In the example environment 600 of FIG. 600 , client computing devices 606 ( 1 ) through 606 (N) can use their respective client modules 620 to connect to each other and / or to (one or more) other external devices in order to participate in a communication session 603 or to contribute activities to the collaborative environment. For example, a first user can use client computing device 606 ( 1 ) to communicate with a second user of another client computing device 606 ( 2 ). When executing client module 620 , the users can share data, which can cause client computing device 606 ( 1 ) to connect to system 602 and / or other client computing devices 606 ( 2 ) through 606 (N) via network 608 .

[0098] The client computing devices 606(1) to 606(N) may use their respective profile modules 622 to generate participant profiles ( Figure 86) and provide participant profiles to other client computing devices and / or devices 610 of system 602. A participant profile may include one or more of an identity of a user or a group of users (e.g., name, unique identifier (“ID”), etc.), user data such as personal data, user data such as location (e.g., IP address, room in a building, etc.) and technical capabilities, etc. The participant profile may be used to register participants for a communication session.

[0099] like Figure 8 As shown, device 610 of system 602 includes server module 630 and output module 632. In this example, server module 630 is configured to receive media streams 634(1) to 634(N) from various client computing devices, such as client computing devices 606(1) to 606(N). As described above, the media streams may include video feeds (e.g., audio and visual data associated with a user), audio data to be output with the presentation of the user's avatar (e.g., an audio-only experience that does not send the user's video data), text data (e.g., a text message), file data and / or screen sharing data (e.g., a document, a slide deck, an image, a video displayed on a display screen, etc.), etc. Thus, server module 630 is configured to receive a collection of various media streams 634(1) to 634(N) (the collection is referred to herein as "media data 634") during real-time viewing of communication session 603. In some scenarios, not all client computing devices participating in communication session 603 provide media streams. For example, the client computing device may be a consuming device or a “listening” device only, such that it only receives content associated with the communication session 603 , but does not provide any content to the communication session 603 .

[0100] In various examples, the server module 630 can select aspects of the media stream 634 to be shared with each of the participating client computing devices 606(1) to 606(N). Thus, the server module 630 can be configured to generate session data 636 based on the media stream 634 and / or pass the session data 636 to the output module 632. The output module 632 can then transmit communication data 639 to the client computing devices (e.g., the client computing devices 606(1) to 606(3) participating in the real-time viewing of the communication session). The communication data 639 can include video, audio, and / or other content data provided by the output module 632 based on content 650 associated with the output module 632 and based on the received session data 636. The content 650 can include the media stream 634 or other shared data, such as an image file, a spreadsheet file, a slide deck, a document, etc. The media stream 634 can include a video component depicting an image captured by the I / O device 626 on each client computer. The content 650 also includes input data from each user, which can be used to control the direction and position of the presentation. The content can also include instructions for sharing data and identifiers for recipients of the shared data. Therefore, the content 650 is also referred to as input data 650 or input 650 in this article.

[0101] As shown, output module 632 transmits communication data 639(1) to client computing device 606(1), and communication data 639(2) to client computing device 606(2), and communication data 639(3) to client computing device 606(3), etc. The communication data 639 sent to the client computing devices may be the same or may be different (e.g., the positioning of the content stream within the user interface may vary from one device to the next).

[0102] In various embodiments, the device 610 and / or the client module 620 may include a GUI presentation module 640. The GUI presentation module 640 may be configured to analyze the communication data 639 for delivery to one or more of the client computing devices 606. Specifically, at the device 610 and / or the client computing device 606, the UI presentation module 640 may analyze the communication data 639 to determine an appropriate manner for displaying the video, image, and / or content on the display screen 629 of the associated client computing device 606. In some embodiments, the GUI presentation module 640 may provide the video, image, and / or content to a presentation GUI 646 that is presented on the display screen 629 of the associated client computing device 606. The presentation GUI 646 may be presented on the display screen 629 by the GUI presentation module 640. The presentation GUI 646 may include the video, image, and / or content analyzed by the GUI presentation module 640.

[0103] In some implementations, the presentation GUI 646 may include multiple sections or grids that may present or include videos, images, and / or content for display on the display screen 629. For example, a first section of the presentation GUI 646 may include a video feed of a presenter or individual, and a second section of the presentation GUI 646 may include a video feed of an individual consuming conference information provided by the presenter or individual. The GUI presentation module 640 may populate the first section and the second section of the presentation GUI 646 in a manner that appropriately mimics an ambient experience that the presenter and individual may share.

[0104] In some embodiments, the GUI presentation module 640 can zoom in or provide a zoomed view of an individual represented by a video feed to highlight the individual's reaction to the presenter, such as facial features. In some embodiments, the presentation GUI 646 can include video feeds of multiple participants associated with a conference, such as a general communication session. In other embodiments, the presentation GUI 646 can be associated with a channel, such as a chat channel, an enterprise team channel, etc. Thus, the presentation GUI 646 can be associated with an external communication session that is different from a general communication session.

[0105] Fig. 9 A diagram showing example components of an example device 700 (also referred to herein as a "computing device") configured to generate data for some user interfaces disclosed herein is shown. Device 700 can generate data that can include one or more portions of a video, image, virtual object, and / or content that can be rendered or included for display on display screen 629. Device 700 can represent one of the devices described herein. Additionally or alternatively, device 700 can represent one of client computing devices 606.

[0106] As shown, device 700 includes one or more data processing units 702, computer-readable media 704, and communication interface 706. The components of device 700 are operably connected, for example, via bus 709, which may include one or more of a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and any of a variety of local, peripheral, and / or independent buses.

[0107] As used herein, data processing unit(s) (such as data processing unit(s) 702 and / or data processing unit(s) 692) may represent, for example, a CPU-type data processing unit, a GPU-type data processing unit, a field programmable gate array ("FPGA"), another type of DSP, or other hardware logic components that may be driven by a CPU in some cases. By way of example and not limitation, illustrative types of hardware logic components that may be used include application specific integrated circuits ("ASICs"), application specific standard products ("ASSPs"), systems on chips ("SOCs"), complex programmable logic devices ("CPLDs"), and the like.

[0108] As used herein, computer-readable media (such as computer-readable media 704 and computer-readable media 694) can store instructions that can be executed by (multiple) data processing units. The computer-readable medium can also store instructions that can be executed by an external data processing unit, such as by an external CPU, an external GPU, and / or can be executed by an external accelerator, such as an FPGA-type accelerator, a DSP-type accelerator, or any other internal or external accelerator. In various examples, at least one CPU, GPU, and / or accelerator is incorporated into the computing device, while in some examples, one or more of the CPU, GPU, and / or accelerator is external to the computing device.

[0109] Computer readable media (also referred to herein as computer readable media) may include computer storage media and / or communication media. Computer storage media may include one or more of volatile memory, non-volatile memory and / or other persistent and / or secondary computer storage media, removable and non-removable computer storage media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Thus, computer storage media that is part of the device or external to the device includes, but is not limited to, random access memory ("RAM"), static random access memory ("SRAM"), dynamic random access memory ("DRAM"), phase change memory ("PCM"), read-only memory ("ROM"), erasable programmable read-only memory ("EPROM"), electrically erasable programmable read-only memory ("EEPROM"), flash memory, compact disk read-only memory ("CD-ROM"), digital versatile disk ("DVD"), optical card or other optical storage medium, magnetic cassette, magnetic tape, magnetic disk storage, magnetic card or other magnetic storage device or medium, solid-state memory device, storage array, network attached storage device, storage area network, hosted computer storage device, or any other storage memory, storage device and / or storage medium that can be used to store and maintain information for access by a computing device. Computer storage media may also be referred to herein as computer-readable storage media, non-transitory computer-readable storage media, non-transitory computer-readable media, or computer storage media.

[0110] In contrast to computer storage media, communication media may embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism. As defined herein, computer storage media does not include communication media. That is, computer storage media itself does not include communication media consisting solely of a modulated data signal, a carrier wave, or a propagating signal.

[0111] The communication interface 706 may represent, for example, a network interface controller ("NIC") or other type of transceiver device for sending and receiving communications over a network. In addition, the communication interface 706 may include one or more cameras and / or audio devices 722 to enable the generation of video feeds and / or still images, etc.

[0112] In the illustrated example, computer-readable medium 704 includes data storage 708. In some examples, data storage 708 includes a data storage, such as a database, a data warehouse, or other type of structured or unstructured data storage. In some examples, data storage repository 708 includes a corpus and / or a relational database having one or more tables, indexes, stored procedures, etc., to enable data access, including, for example, one or more of a Hypertext Markup Language ("HTML") table, a Resource Description Framework ("RDF") table, a Web Ontology Language ("OWL") table, and / or an Extensible Markup Language ("XML") table.

[0113] The data store 708 may store data for the operation of processes, applications, components, and / or modules stored in the computer-readable medium 704 and / or executed by the (one or more) data processing units 702 and / or (one or more) accelerators. For example, in some examples, the data store 708 may store session data 710 (e.g., Figure 8 The data store 708 may also include context data 714, such as content including video, audio, or other content for presentation and display on one or more of the display screens 629. The hardware data 711 may define aspects of any device, such as multiple display screens of a computer. The context data 714 may define any type of activity or status associated with individual users 10A-10L, each associated with each of the multiple video streams 634. For example, the context data may define the level of a person in an organization, how each person's level relates to the level of other people, the performance level of a person, or any other activity or status information that may be used to determine the location of a person's rendering within a virtual environment. This contextual information can also be fed into any model to help emphasize keywords spoken by a person of a certain level, highlight the UI when background voices of a person of a certain level are detected, or change the emotion display in a specific way when a person of a certain level is detected to have a certain emotion.

[0114] Alternatively, some or all of the above data may be stored on a separate memory 716 on one or more data processing units 702, such as a memory on a single board, a CPU-type processor, a GPU-type processor, an FPGA-type accelerator, a DSP-type accelerator, and / or another accelerator. In this example, the computer-readable medium 704 also includes an operating system 718 and an application programming interface 710 (API) configured to disclose functions and data of the device 700 to other devices. In addition, the computer-readable medium 704 includes one or more modules, such as a server module 730, an output module 732, and a GUI presentation module 740, although the number of modules shown is only an example and the number may vary. That is, the functionality described herein in conjunction with the illustrated modules may be performed by a smaller number of modules or a larger number of modules on one device or spread across multiple devices.

[0115] Finally, although various configurations have been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the accompanying representations is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claimed subject matter.

Claims

1. A method for generating a visual indicator for distinguishing an active speaker from a user of a communication session displayed as a 3D representation, the method being configured for execution on a system, the method comprising: causing display of a first user interface arrangement including individual renderings of three-dimensional representations of a plurality of users participating in the communication session, wherein each of the three-dimensional representations has an independent position and orientation within a three-dimensional environment, the independent positions and orientations being each controlled by a control input provided by an associated user of the plurality of users, the first user interface arrangement further comprising renderings of a subset of users in a 2D format, the renderings of the subset of users being generated by a 2D device associated with the subset of users; receiving input from the plurality of users identifying a user as the active speaker, wherein the user is identified as the active speaker by detecting speech input received by a microphone associated with the user, the microphone generating an audio stream received for the communication session; determining that the user is a member of the users rendered in the 3D representation; In response to the user being identified as the active speaker and being the member of users rendered in a 3D representation, causing the first user interface arrangement to transition to a second user interface arrangement including the three-dimensional representation of the plurality of users including the user and a second additional rendering in a 2D format of the user, wherein: the first user interface arrangement does not simultaneously display the 3D representation of the user and the second additional rendering of the 2D format of the user, and The second user interface arrangement simultaneously displays the second additional rendering of the user and the 3D representation of the user positioned within the three-dimensional environment.

2. The method according to claim 1, wherein: The second user interface arrangement includes a first area reserved for rendering of the three-dimensional environment and a second area reserved for rendering of an active speaker of the communication session, the second area including a 2D rendering of a video stream of a user who qualifies as the active speaker, wherein the second rendering of the user is at least partially displayed within the second area.

3. The method according to claim 1, wherein: The second user interface arrangement includes a first area reserved for a rendering of the three-dimensional environment and a second area reserved for a rendering of an active speaker, the active speaker being eligible to enter an overflow queue for a user, the overflow queue being secondary to a primary queue for the user, wherein the second rendering of the user is displayed at least partially within the second area.

4. The method according to claim 1, wherein: The second user interface arrangement includes a first area reserved for a rendering of the three-dimensional environment and a second area reserved for a rendering of an active speaker of the communication session, the second area at least partially overlapping the first area, wherein the second rendering of the user is at least partially displayed within the second area.

5. The method according to claim 1, wherein: The system controls the transition of the first user interface arrangement to the second user interface arrangement based on the size of the rendering of the three-dimensional environment, wherein when the size of the rendering of the three-dimensional environment is less than a size threshold, the system prevents the transition of the first user interface arrangement to the second user interface arrangement, and wherein when the size of the rendering of the three-dimensional environment is greater than the size threshold, the system allows the transition of the first user interface arrangement to the second user interface arrangement.

6. The method according to claim 1, wherein: The system controls the transition of the first user interface arrangement to the second user interface arrangement based on the title or role of the user, wherein if the title or the role of the user does not satisfy one or more criteria, the system prevents the transition of the first user interface arrangement to the second user interface arrangement, wherein if the title or the role of the user satisfies one or more criteria, the system allows the transition of the first user interface arrangement to the second user interface arrangement.

7. The method of claim 1, wherein the second user interface arrangement comprises a first area reserved for rendering of the three-dimensional environment and a second area reserved for rendering of an active speaker of the communication session, wherein: The second area is in a grid format having individual sections for each user rendering, wherein transitioning of the first user interface arrangement to the second user interface arrangement comprises adding a new grid section for the second rendering of the user.

8. A system for generating a visual indicator for distinguishing an active speaker from a user of a communication session displayed as a 3D representation, the system comprising: one or more processing units; as well as A computer-readable storage medium having computer-executable instructions encoded thereon to cause the one or more processing units to perform the following operations: causing display of a first user interface arrangement including individual renderings of three-dimensional representations of a plurality of users participating in a communication session, wherein each of the three-dimensional representations has an independent position and orientation within a three-dimensional environment, the independent positions and orientations being each controlled by a control input provided by an associated user of the plurality of users, the first user interface arrangement further comprising renderings in a 2D format of a subset of the users, the renderings of the subset of users being generated by a 2D device associated with the subset of users; receiving input from the plurality of users identifying a user as the active speaker, wherein the user is identified as the active speaker by detecting speech input received by a microphone associated with the user, the microphone generating an audio stream received for the communication session; determining that the user is a member of the users rendered in the 3D representation; In response to the user being identified as the active speaker and being the member of users rendered in a 3D representation, causing the first user interface arrangement to transition to a second user interface arrangement including the three-dimensional representation of the plurality of users including the user and a second additional rendering in a 2D format of the user, wherein: the first user interface arrangement does not simultaneously display the 3D representation of the user and the second additional rendering of the 2D format of the user, and The second user interface arrangement simultaneously displays the second additional rendering of the user and the 3D representation of the user positioned within the three-dimensional environment, wherein the second rendering of the second user interface arrangement serves as the visual indicator for distinguishing the user as the active speaker from the users of the communication session displayed as 3D representations.

9. The system according to claim 8, wherein: The second user interface arrangement includes a first area reserved for rendering of the three-dimensional environment and a second area reserved for rendering of an active speaker of the communication session, the second area including a 2D rendering of a video stream of a user who qualifies as the active speaker, wherein the second rendering of the user is at least partially displayed within the second area.

10. The system according to claim 8, wherein: The second user interface arrangement includes a first area reserved for a rendering of the three-dimensional environment and a second area reserved for a rendering of an active speaker, the active speaker being eligible to enter an overflow queue for a user, the overflow queue being secondary to a primary queue for the user, wherein the second rendering of the user is displayed at least partially within the second area.

11. The system according to claim 8, wherein: The second user interface arrangement includes a first area reserved for a rendering of the three-dimensional environment and a second area reserved for a rendering of an active speaker of the communication session, the second area at least partially overlapping the first area, wherein the second rendering of the user is at least partially displayed within the second area.

12. The system according to claim 8, wherein: The system controls the transition of the first user interface arrangement to the second user interface arrangement based on the size of the rendering of the three-dimensional environment, wherein when the size of the rendering of the three-dimensional environment is less than a size threshold, the system prevents the transition of the first user interface arrangement to the second user interface arrangement, and wherein when the size of the rendering of the three-dimensional environment is greater than the size threshold, the system allows the transition of the first user interface arrangement to the second user interface arrangement.

13. The system according to claim 8, wherein: The system controls the transition of the first user interface arrangement to the second user interface arrangement based on the title or role of the user, wherein if the title or the role of the user does not satisfy one or more criteria, the system prevents the transition of the first user interface arrangement to the second user interface arrangement, wherein if the title or the role of the user satisfies one or more criteria, the system allows the transition of the first user interface arrangement to the second user interface arrangement.

14. The system of claim 8, the second user interface arrangement comprising a first area reserved for rendering of the three-dimensional environment and a second area reserved for rendering of an active speaker of the communication session, wherein: The second area is in a grid format having individual sections for each user rendering, wherein transitioning of the first user interface arrangement to the second user interface arrangement comprises adding a new grid section for the second rendering of the user.

Images