Automation of audio and viewing angles for focusing to related activities of communication session

Abstract

Techniques disclosed herein enable a system to direct the attention of a user to related activities of a communication session displayed in a 3D environment. The system may control a viewing angle of view of the 3D environment to focus on a particular avatar associated with a user viewing the activity. The system may also control audio signals from remote users of the communication session to enable viewers to focus on related discussions and other audio content. For example, when a particular group of remote users controls their avatars to view the user's avatar or mention the user's name in a session, the system may obscure views of other avatars to focus on the particular group of remote users. By guiding visual and audio views, the system may help the user browse activities of a large group of participants communicating using a 3D representation of the 3D environment.

Description

Background Art

[0001] The use of meta-verse environments in online meeting applications is becoming ubiquitous. Participants in online meetings now meet in three-dimensional virtual environments and share content within those virtual environments. Despite many benefits compared to other forms of collaboration, using 3D environments to share content can come with many disadvantages.

[0002] One of the main problems with using a metaverse environment for online meeting applications is that there may be scenarios where it may be difficult for participants in a meeting to identify relevant user activities. Some systems that support 3D environments may allow many users to participate. Some environments allow hundreds of users. For certain types of events, such as meetings, it may be difficult for users to identify specific conversations and specific people participating in activities of interest. This can lead to many inefficiencies because it may require users to search for related activities. This can be a difficult task considering that traditional search methods are not available in real-time conversations. When three-dimensional environments are involved, navigation tools are not always optimized for searching for relevant activities, and it may be difficult to browse large groups of people to search for relevant activities.

[0003] These shortcomings may lead to inefficient interactions between computing devices and users, particularly during communication sessions. In addition, the above shortcomings of existing systems may lead to loss of user engagement. Computing devices that do not promote user engagement or worse lead to loss of user engagement and suboptimal interactions may lead to production losses and inefficiencies with respect to multiple computing resources. For example, when a user becomes tired or disengaged, when shared content is missed or ignored, the user may need to refer to other resources, such as documents, or use other forms of communication. When a viewer misses an important point or clue during a real-time meeting, the missed content may need to be resent. Such activities may lead to inefficient or repeated use of networks, processors, memory, or other computing resources. Therefore, there is a continuous need to develop improvements to help make the user experience of a communication system more attractive and more like or better than a face-to-face meeting. Summary of the invention

[0004] The technology disclosed herein enables the system to direct the user's attention to the relevant activities of the communication session displayed in the 3D environment. The system can control the viewing angle to focus on the specific avatar related to the user watching the activity. The system can also control the audio signal from the remote user of the communication session so that the viewer can focus on the relevant discussion and other audio content. For example, when the remote user group controls their avatar to perform a gesture to identify the selected user, such as the remote user group mentions the name of the selected user in the conversation, or controls their avatar to view the avatar of the selected user, the system can cover the view of other avatars to focus on the avatar of the remote user group. The system can also increase the volume of the audio signal from the remote user group and amplify the avatar of the remote user group. The focused view enables the selected user to easily see and hear the relevant activities of the specific remote user group. The system can focus on one or more members of the specific remote user group in response to any type of activity (such as file sharing) of the remote user identifying the user. By guiding the visual and audio perspective of the 3D environment, the system can help users browse the activities of a large group of participants who are collaborating using the 3D representation of the 3D environment.

[0005] In one illustrative example, the system may initially display a first user interface arrangement on a display device pointing to a user (e.g., a viewer). The first user interface arrangement may include individual renderings of 3D representations of remote users participating in a communication session with the user. Each 3D representation may be an avatar of the remote user, each avatar having a controllable position and orientation within a 3D environment. The system may then monitor user activity of the communication session to identify at least one input from a remote user controlling one of the displayed avatars. For example, the input may indicate that the remote user is controlling their avatar to direct a gesture toward the user. For example, the remote user may direct their avatar to view the user's avatar or move toward the user's avatar. Any other activity of the remote user identifying the user may be used to trigger the user interface and audio conversion described herein. For example, when a remote user group identifies the user in a conversation or when the remote user group begins to share content with the user, such activity may cause the system to trigger the user interface and audio conversion described herein. The system may also track the identity of the remote user performing the triggering activity. For illustrative purposes, the remote user performing the triggering activity is referred to herein as a subset of users participating in the communication session.

[0006] When the system detects input that identifies a specific user and triggers a specific activity of a conversion of a user interface or an audio signal, the system converts a first user interface arrangement including individual renderings of 3D representations of users participating in a communication session to a second user interface arrangement that focuses on a subset of one or more 3D representations associated with a remote user subset in these 3D representations. The system can focus on the user subset by obscuring or blurring the representations of other users. The system can also focus on the user subset by increasing the clarity level of the avatars of the user subset or zooming the perspective view focused on the avatars associated with the user subset. The system can also focus on the user subset by increasing the volume of the audio signal received from the computing device of the user subset. The system can also focus on the user subset by reducing the volume of other remote users other than the user subset. The system can control the audio of the remote user in any way that allows the user to hear the difference between the user who is part of the user subset and other users who are not part of the user subset of the communication session.

[0007] The present technology disclosure provides many technical advantages. For example, by promoting user participation and helping users find relevant activities and related information, especially in communication systems, users can exchange information more efficiently. This helps to mitigate the occurrence of lost or overlooked shared content. This can reduce the occurrence of users needing to resend information. More efficient communication of shared content can also help avoid the need for external systems (such as mobile phones and other messaging platforms for texting). This can help reduce the reuse of networks, processors, memory, or other computing resources, especially when extended meetings or additional meetings can be avoided.

[0008] By reading the following detailed description and viewing the associated drawings, features and technical advantages other than those explicitly described above will become apparent. This summary is intended to introduce a set of concepts in a simplified form, which will be further described in the detailed description below. This summary is not intended to identify the key features or essential features of the claimed subject matter, nor is it intended to be used to help determine the scope of the claimed subject matter. For example, as allowed by the above context and throughout the document, the term "technology" may refer to systems, methods, computer-readable instructions, modules, algorithms, hardware logic and / or operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The detailed description is described with reference to the accompanying drawings. In the drawings, the leftmost digit of a reference number identifies the figure in which the reference number first appears. The same reference numbers in different figures indicate similar or identical items. References to individual items in a plurality of items may use reference numbers with letters of the alphabetical sequence to refer to each individual item. General references to items may use the specific reference number without the alphabetical sequence.

[0010] Figure 1A A first user interface arrangement showing a plurality of users is shown.

[0011] Figure 1B A second user interface arrangement is shown that uses a blur feature to focus on a subset of a plurality of users.

[0012] Figure 1C Another user interface arrangement is shown that uses a blur feature to focus on another subset of multiple users.

[0013] Figure 2A A first user interface arrangement showing a plurality of users is shown.

[0014] Figure 2B A second user interface arrangement is shown that uses a perspective modification feature on a 3D rendering to focus on a subset of a plurality of users.

[0015] Figure 2C Another user interface arrangement is shown that uses a perspective modification feature on a 3D rendering to focus on another subset of multiple users.

[0016] Figure 3A A first user interface arrangement showing a plurality of users is shown.

[0017] Figure 3B A second user interface arrangement focusing on a subset of the plurality of users is shown adding a 2D video rendering and using a blur feature on the 3D rendering.

[0018] Figure 4A A first user interface arrangement showing a plurality of users is shown.

[0019] Figure 4B A second user interface arrangement is shown that focuses on a subset of multiple users by adding 2D video rendering and using a perspective modification feature.

[0020] Figure 5A A scene is shown in which the user's avatar is directed away from the selected user.

[0021] Figure 5B A scene is shown where the user's avatar is directed away from a selected user and then moves toward the selected user.

[0022] Figure 5C A scene is shown in which the system controls the viewing perspective of a selected user in response to avatars of other users directing the field of view toward the selected user's avatar.

[0023] Figure 6 Additional technical details regarding UI rendering disclosed herein are shown.

[0024] Figure 7is a flow chart showing aspects of a routine for controlling a viewing perspective of a three-dimensional environment to focus on relevant activities of users participating in a communication session.

[0025] Figure 8 is a computer architecture diagram showing an illustrative computer hardware and software architecture for a computing system capable of implementing aspects of the techniques and technologies presented herein.

[0026] Fig. 9 is a computer architecture diagram illustrating a computing device architecture of a computing device capable of implementing aspects of the techniques and technologies presented herein. DETAILED DESCRIPTION

[0027] Figure 1A and Figure 1B An example of a UI transition that changes the viewing perspective of a 3D environment to focus on the relevant activities of users participating in a communication session is shown. A communication session can be managed by a system 100 including multiple computers 11, each computer corresponding to multiple users 10. In this example, a first user 10A (Reta Taylor) is associated with a first computer 11A, a second user 10B (Miguel Silva) is associated with a second computer 11B, a third user 10C (Bryan Bright) is associated with a third computer 11C, a fourth user 10D (MJ Price) is associated with a fourth computer 11D, a fifth user 10E (Bruno Zhaos) is associated with a fifth computer 11E, a sixth user 10F (Serena Davis) is associated with a sixth computer 11F, a seventh user 10G (Krystal McKinney) is associated with a seventh computer 11G, an eighth user 10H (Jessica Kline) is associated with an eighth computer 11H, a ninth user 10I (Kat Larsson) is associated with a ninth computer 11I, and a tenth user 10J (Traci Isaac) is associated with a tenth computer 11J. The users may also be referred to as "User A", "User B", etc., respectively. Other users (e.g., User K and User L) are also participants in the communication session. Each user may be displayed as a two-dimensional image in the user interface, or each user may be displayed as a three-dimensional representation 251 in the user interface. The 3D representation may be a static model or a dynamic model that animates in real time in response to user input. Although this example shows a user interface in which the users are displayed as 3D representations, it will be appreciated that the techniques disclosed herein may be applied to other forms of images, such as a 2D image of each user.

[0028] The computer may be in the form of a desktop computer, a head mounted display unit, a tablet device, a mobile phone, etc. The system may generate a user interface that presents aspects of the communication session to each user. In this example, the first user interface arrangement 101A may include multiple renderings of each user 10. The rendering may include a rendering of a two-dimensional (2D) image, which may include a picture of the user or a real-time video feed. The first user interface 101A may also include a rendering of a three-dimensional 3D representation 251, which may include an avatar located within the 3D virtual environment 200. In this particular example, the user interface 101 includes a 3D rendering of a representation 251A of the first user 10A, a 3D rendering of a representation 251B of the second user 10B, a 3D rendering of a representation 251C of the third user 10C, a 3D rendering of a representation 251D of the fourth user 10D, and a 3D rendering of a representation 251E of the fifth user 10E. The first user interface arrangement 101A may also include 2D renderings of these and other users. In this example, the first user interface arrangement 101A is displayed on a display device in communication with a tenth computing device 11J of a tenth user 10J.

[0029] In this example, the system executes from Figure 1A UI to Figure 1B . The system causes display of a first user interface arrangement 101A on a display device 629 associated with the user 10J. The first user interface arrangement 101A includes individual renderings of three-dimensional representations 251A-251E of multiple users 10A-10E participating in a communication session with the user 10J. The first user interface arrangement 101A may also include 2D renderings of image files or real-time streams of these multiple users or other users. The individual three-dimensional representations 251A-251E have independent positions and independent orientations within the three-dimensional environment 200, and the independent positions and the independent orientations are each controlled by input data 650 provided by an associated user of the multiple users 10A-10E. Figure 1A An example of a first user interface arrangement 101A is shown in . In this example, the first user interface arrangement 101A has a rendering of a 3D avatar 251 in a 3D virtual environment, and the viewing user is user J, also referred to herein as selected user 10J.

[0030] The system monitors user activity of the communication session to identify at least one remote user that provides input or identifies user 10J. For example, the system may monitor input activity to identify when a threshold number of remote users 10A-10E state the name of user J. In another example, the system may monitor input activity to identify when a threshold number of remote users 10A-10E control their avatars to view the avatar of user J. In yet another example, the system may monitor input activity to identify when a threshold number of remote users 10A-10E share data, such as files or meeting content, with user J.

[0031] In some embodiments, one or more computers may monitor input data 650 from multiple users 10A-10E participating in a communication session 604 with user 10J. The system is configured to take one or more actions when the system detects input data 650 identifying or referencing user J in an avatar's gesture, input data 650 identifying user J in one or more operations for sharing content, and / or input data 650 identifying user J in one or more forms of communication where user J is identified in a verbal conversation, text message, content, etc. For example, when a remote user specifically mentions user J by name in a Word document using an at-mention, the system may trigger one or more actions for controlling user J's visual or audio perspective to draw attention to the remote user's activity.

[0032] The system can determine a user subset from multiple users. In one example, if the first user 10A controls their avatar to view user J or if the first user 10A mentions the name of user J in any document or communication, the system identifies the first user 10A as part of the user subset. Input data 650 identifying user 10J is received from one or more computing devices 11A-11B associated with one or more users (e.g., user A and user B), and the system determines that these users are part of the user subset. In another illustrative example, if the first user 10A controls their avatar to view user J, the first user is identified as part of the user subset. In yet another example, the user subset can also be associated with the user who provides the input data. For example, if the first user 10A controls their avatar to view user J, and the first user 10A has an avatar within a threshold distance of other avatars of other users (e.g., the second user 10B), the other users (e.g., the second user 10B) can also be identified as part of the user subset. In another example, if the first user 10A controls their avatar to view or share content with user J, and the first user 10A has an avatar within a threshold distance of other avatars of other users or the other users are in conversation with the first user 10A, the system can identify the other users as part of the user subset. Once the user subset is identified, the system causes a transition of the user interface.

[0033] In response to input data 650 identifying user 10J, the system may Figure 1A The first user interface arrangement 101A shown is converted to Figure 1BThe second user interface arrangement 101B shown. The transition may include a first display of a first user interface arrangement 101A, the first user interface arrangement 101A including individual renderings of three-dimensional representations 251A-251E of a plurality of users 10A-10E participating in the communication session 604, to a second user interface arrangement 101B focused on a subset of one or more of the three-dimensional representations 251A-251B associated with a subset of users 10A-10B. The focus on the three-dimensional representations 251A-251B of the subset of users 10A-10B generates a visual focus on the 3D representations 251A-251B of the subset of users. In this example, the focus may include modifying display properties to give the appearance that the lens is focusing on the 3D representations 251A-251B of the subset of users. This may include an increased level of clarity relative to the lines of the representations 251A-251B or a filter that removes any aberrations or graphical elements that distort the representations 251A-251B. Focusing on the representations 251A-251B also includes rendering other representations 251C-251E of other users of the plurality of users 10C-10E such that they appear out of focus or blurred or obscured.

[0034] The system may also focus on audio signals generated by computers 11A-11B associated with a subset of users. Audio focusing may include controls that distinguish audio generated by computers of a subset of users from audio generated by computers of other users 10C-10E. For example, the system may increase the volume associated with the microphone of each user 10A-10B and turn down the microphones of other users 10C-10E. Simultaneously or alternatively, the system may also increase the focus level on three-dimensional representations 251A-251B associated with the subset of users 10A-10B, and reduce the focus on the display of representations 251C-251E of other users 10C-10E in the plurality of users or block the display of representations 251C-251E. The shifted focus may also include zooming in on three-dimensional representations 251A-251B associated with the subset of users 10A-10B.

[0035] In addition to Figure 1A The first user interface arrangement 101A shown is converted to Figure 1B In addition to or in lieu of the second user interface arrangement 101B shown, the system may also adjust the volume in response to input to differentiate audio signals from computing devices 11A-11B of a subset of users. For example, in response to input from computing devices 11A-11B, where the input identifies a selected user 10J, the system may increase the volume of the audio stream of the user associated with computing device 11A-11B. In response to the input, the system may decrease the volume of the audio streams generated by computing devices of other users 10C-10E of the plurality of users.

[0036] Figure 1B and Figure 1C Another example of a transformation that may occur when another subset of users 10C-10E provides one or more inputs identifying a selected user 10J is shown. In response to such input identifying user 10J, the system may Figure 1B The second user interface arrangement 101B shown is converted to Figure 1C The transition may start with display of a second user interface arrangement 101B including individual renderings of three-dimensional representations 251A-251B of the first subset of users 10A-10B, and transition to a third user interface arrangement 101C focused on the second subset of users 10C-10E displayed as representations 251C-251E. Figure 1C As shown, after the second transition, the focus on the first subset of one or more of the three-dimensional representations 251A-251B is reduced, for example, the three-dimensional representations 251A-251B of the first user subset are no longer in focus view, for example, they are blurred, dimmed, obscured or otherwise removed. Moreover, in a second transition to the third user interface arrangement 101C, the focus is then transferred to the 3D representations 251C-251E of the second user subset 10C-10E. Additionally or simultaneously, the volume of the audio signal generated by the computing device of the second user subset 10C-10E can also be increased, and the audio signal of the first user subset 10A-10B is reduced.

[0037] Figure 2A-2C Another example of a user interface transition that may occur when a trigger input is received is shown. In this example, the user interface transition includes an adjustment to the viewing perspective (e.g., zoom level) that allows a user (e.g., user J) to easily see relevant content and activities. The first user interface arrangement 101A is similar to the above description of Figure 1A The rendering described is similar to the rendering start.

[0038] In response to input data 650 identifying user 10J, the system may Figure 2A The first user interface arrangement 101A shown is converted to Figure 2BThe second user interface arrangement 101B shown. The transition may include a first display of a first user interface arrangement 101A, the first user interface arrangement 101A including individual renderings of three-dimensional representations 251A-251E of a plurality of users 10A-10E participating in the communication session 604, to a second user interface arrangement 101B focusing on a subset of one or more of the 3D representations 251A-251B associated with a subset of users 10A-10B. Focusing on a subset of one or more of the three-dimensional representations 251A-251B generates a visual focus on the three-dimensional representations 251A-251B of the subset of users. In this example, the visual focus includes changing the perspective, such as causing the camera to focus on the 3D representations 251A-251B of the subset of users 10A-10B. The focus on the subset of users also includes removing or otherwise obscuring representations 251C-251E of other users 10C-10E of the plurality of users.

[0039] The system may also focus on audio signals generated by computers 11A-11B associated with a subset of users. Audio focusing may include controls that distinguish audio generated by computers of the subset of users from audio generated by computers of other users 10C-10E. For example, the system may increase the volume associated with the microphone of each user 10A-10B and turn down the microphones of other users 10C-10E. Simultaneously or alternatively, the system may also increase the focus level on three-dimensional representations 251A-251B associated with the subset of users 10A-10B and reduce the focus on or block the display of representations 251C-251E of other users 10C-10E in the plurality of users.

[0040] In addition to Figure 2A The first user interface arrangement 101A shown is converted to Figure 2B In addition to or in lieu of the second user interface arrangement 101B shown, the system may adjust the volume in response to input to differentiate audio signals from computing devices 11A-11B of a subset of users. For example, in response to input from computing devices 11A-11B, where the input identifies a selected user 10J, the system may increase the volume of the audio stream of the user associated with computing device 11A-11B. In response to the input, the system may decrease the volume of the audio streams generated by computing devices of other users 10C-10E of the plurality of users.

[0041] Figure 2B and Figure 2C Another example of a transformation that may occur when another subset of users 10C-10E provides one or more inputs identifying a selected user 10J is shown. In response to such input identifying user 10J, the system may Figure 2BThe second user interface arrangement 101B shown is converted to Figure 2C The transition may start with display of a second user interface arrangement 101B including individual renderings of three-dimensional representations 251A-251B of the first subset of users 10A-10B, and transition to a third user interface arrangement 101C focused on the second subset of users 10C-10E displayed as representations 251C-251E. Figure 2C As shown, after this second transition, the focus on the first subset of one or more of the three-dimensional representations 251A-251B is removed, for example, the three-dimensional representations 251A-251B of the first user subset are no longer in view. Moreover, in a second transition to the third user interface arrangement 101C, the focus is then transferred to the 3D representations 251C-251E of the second user subset 10C-10E. This can include changing the viewing perspective to appear as if the virtual camera is pointing at the avatars of the second user subset. Additionally or simultaneously, the volume of the audio signal generated by the computing device of the second user subset 10C-10E can also be increased, and the audio signal of the first user subset 10A-10B is reduced.

[0042] Figure 3A-3B Another example of a user interface transition that may occur when a trigger input is received is shown. In this example, the user interface transition includes an adjustment of the focus level of a selected rendering that allows a user (e.g., user J) to easily view related content and activities. The first user interface arrangement 101A is similar to the above description of Figure 1A The 3D rendering similar to the rendering described above begins. The first user interface arrangement 101A also includes a 2D rendering of at least one subset of users displayed as a rendering 251. For example, the user interface may include a video rendering 151A of the first user 10A, a video rendering 151B of the second user 10B, a video rendering 151C of the third user 10C, a video rendering 151D of the fourth user 10D, and a rendering of the viewing user (user J). As described below, when the system focuses on a selected 3D representation 251 or other participant image, the system can display a 2D rendering of the video streams of those selected users. In this example, the first user interface arrangement 101A is displayed on a display device that communicates with the tenth computing device 11J of the tenth user 10J.

[0043] In response to input identifying a selected user, the system causes Figure 3A The first user interface shown is arranged to Figure 3B The transition to the second user interface arrangement shown. Figure 3B In the transition of the second user interface arrangement shown, the system uses the above Figure 1CSpecifically, the system focuses on a selected subset of users, in this case users 10C-10E, by focusing on the associated renderings 251C-251E. In addition to focusing on the conversion of the 3D representation, the system also modifies the 2D rendering of the video stream produced by each corresponding user. In this example, Figure 3B The second user interface arrangement shown includes 2D renderings of at least some of the users in the subset of users. For example, because Bryan, MJ, and Bruno are controlling avatars or participating in activities that meet the criteria or pre-set conditions, the system can focus on their 3D representations and, in addition, the system can remove the 3D representations of the avatars. Figure 3A Other 2D images shown, and display 2D images of Bryan, MJ and Bruno, which 2D images may be images of real-time video data. In some embodiments, the second user interface arrangement 101B may also include a "me" view, which includes a 2D rendering of a user (user J) viewing the user interface.

[0044] Figure 4A-4B Another example of a user interface transition that can occur when a trigger input is received is shown. In this example, the user interface transition includes an adjustment of the viewing perspective (e.g., zoom level) that allows a user (e.g., user J) to easily see related content and activities. Figure 4A The first user interface arrangement 101A is similar to the above Figure 1A The rendering described starts similar to the 3D rendering. Figure 4A The first user interface arrangement 101A also includes a 2D rendering of at least one subset of users displayed as a rendering 251. For example, the user interface may include a video rendering 151A of the first user 10A, a video rendering 151B of the second user 10B, a video rendering 151C of the third user 10C, a video rendering 151D of the fourth user 10D, and a rendering of the viewing user (user J). Typically, the 2D renderings 151 of the subset of users can track the participants displayed as representations. As described below, when the system focuses on a selected 3D representation 251 or other participant image, the system can display 2D renderings of the video streams of those selected users. In this example, the first user interface arrangement 101A is displayed on a display device that communicates with the tenth computing device 11J of the tenth user 10J.

[0045] In response to input identifying a selected user (User J), the system causes Figure 4A The first user interface shown is arranged to Figure 4B The transition to the second user interface arrangement shown. Figure 4B In the transition of the second user interface arrangement shown, the system uses the above Figure 2CSpecifically, the system focuses on a selected subset of users, in this case users 10C-10E, by zooming in on the associated renderings 251C-251E and removing the renderings of other users. In addition to focusing on the transformation of the 3D representation, the system also modifies the 2D rendering of the video stream produced by each respective user. In this example, Figure 4B The second user interface arrangement shown includes 2D renderings of at least some of the users in the subset of users. For example, because Bryan, MJ, and Bruno are controlling avatars or participating in activities that meet the criteria or pre-set conditions, the system can focus on their 3D representations and, in addition, the system can remove the 3D representations of the avatars. Figure 4A The other 2D images shown, and display 2D images of Bryan, MJ and Bruno, may be images of real-time video data.

[0046] Figure 5A-Figure 5C A scene is shown where the system controls the viewing perspective of a selected user in response to the avatars of other users directing the field of view towards the avatar of the selected user.These figures show a top view of a 3D environment 200. Figure 5A A first state of the 3D environment is shown where the avatars of user C, user D, and user E are viewing each other with their fields of view not being directed toward a selected user (user J). User B is also in a first state with a viewing area directed toward a selected user.

[0047] like Figure 5B As shown, the second state of the 3D environment shows that user C and user D have rotated to direct their fields of view toward the selected user. When the system detects this type of gesture directed toward the selected user (e.g., a gesture in which the user provides input to move the avatar or change the orientation of the avatar), the system can interpret this type of input as a triggering input to cause any user interface transition described herein. In some embodiments, when a group of users are within a threshold distance of each other, the system can cause a transition of the user interface only when a threshold number of avatars in the group perform a gesture that identifies or is directed toward the selected user. In this example, if the threshold number of users is 2 users, and two of the three users in the group perform a gesture that identifies the selected user, the system can cause one of the user interface transitions described herein. Figure 5C An example of the selected user's avatar after a user interface transition is shown. In this case, the avatar of the selected user (user J) may also rotate or otherwise focus on the users associated with the triggering input (user C and user D (and user E)). This also conveys to user C, user D, and user E that the system has automatically provided a gesture for the transition to the selected user. This signal allows other users to notice that the selected user is engaging with their activities.

[0048] Figure 6 Additional technical details about UI rendering disclosed herein are shown. In some embodiments, upon receiving input for causing a UI transition as described herein, the system may determine the position and orientation of the 3D representation of the selected user. For example, in Figure 5C In the example shown, the system may have to move or rotate the avatar of the selected user when the selected user must rotate to demonstrate their level of engagement with the users providing input (e.g., User C and User D). This means that the virtual object 351J may have to have a position and orientation that allows the corresponding avatar to view other virtual objects.

[0049] Figure 6 Also shown are aspects of a system configured to implement the technology disclosed herein. For illustrative purposes, a rendering of a 2D image file or a rendering of a 2D image of a user can be generated by a 2D rendering engine 551 that receives 2D image data 310 (e.g., an image file). The rendering of a 2D image file can include a 2D environment (e.g., the background of an image) and a 2D object (e.g., an image of a person or avatar). An image file (e.g., image data 310) can have pixels arranged in two dimensions, such as pixels arranged in a two-dimensional coordinate system (x, y). The data may also be referred to herein as a two-dimensional model based on a two-dimensional coordinate system. Each part of the image may be a pixel or any other geometric shape, such as a triangle. For example, a group of pixels or triangles may be used to generate a rendering of a real-time video image of a user's two-dimensional avatar or a person.

[0050] A two-dimensional environment with multiple 2D images of participants in a communication session is also referred to herein as a "grid environment". Image data or a communication data stream may define a two-dimensional environment or a two-dimensional object, and the two-dimensional environment may be presented on a display screen. Rendering may be referred to herein as a two-dimensional rendering of a two-dimensional environment or a two-dimensional rendering of a two-dimensional object. This is also referred to herein as "rendering of a two-dimensional image".

[0051] For illustrative purposes, a rendering of a 3D model or a rendering of a 3D representation of a user can be generated by accessing 3D model data 320 (e.g., a 3D model) by a 3D rendering engine 552. The 3D model may include parameters defining the 3D environment 200, such as a model of a room, and parameters defining 3D objects, such as the size, shape, and position data of a representation 351 of a user or other virtual object. A three-dimensional environment is a computing environment model based on a three-dimensional coordinate system. The properties of a three-dimensional environment and a three-dimensional object in the three-dimensional environment are based on components located within a three-dimensional coordinate system (x, y, z). Each component can be a triangle or any other geometric shape. Each component can have a position (e.g., a position in a three-dimensional coordinate system) and an orientation (e.g., the direction in which the triangle points). For example, a set of triangles can be used to generate a rendering of a three-dimensional avatar of a user or a three-dimensional rendering of a three-dimensional object.

[0052] The three-dimensional environment is also referred to as an "immersive environment" in this article. Model data or a three-dimensional model can be included in the communication data stream, and the model data can define the three-dimensional environment. The three-dimensional environment can be based on a three-dimensional coordinate system. When the rendering engine 552 generates a 3D rendering from the 3D model, the rendering is generated from a reference point in the environment (e.g., a perspective with a position relative to the virtual environment). For illustrative purposes, the reference point is also referred to as a virtual camera 350 in this article. The camera can have a field of view that is used to generate a rendering of a 3D environment or a 3D object based on the position of the virtual camera 350. The rendering of a three-dimensional object in the three-dimensional environment is based on the position and orientation of the three-dimensional object and the position of the virtual camera 350.

[0053] In some embodiments, a two-dimensional image may be displayed within a three-dimensional environment. This may occur, for example, when a communication system receives a two-dimensional video stream of a user, but the participant receiving the video stream is viewing a 3D environment using an HMD. This may cause the system to display the image of the user as if they were appearing on a virtual television on a wall of the virtual environment. This is referred to herein as a two-dimensional rendering of the user within a three-dimensional environment. This may include Figure 2A A third user 10C is shown as rendering 151C.

[0054] In some embodiments, a three-dimensional environment and a three-dimensional object defined by a three-dimensional model can be displayed as a two-dimensional rendering. For example, when a communication session involves a user interface that displays a two-dimensional image, such as when Teams is in grid mode (GridMode), this can happen. While in this mode, the system may need to display images of users interacting in a 3D environment. In this case, a 2D image of the 3D environment is displayed from a specific position (e.g., a virtual camera position), and the 2D image is displayed within one of the grids. This rendering may be referred to as a two-dimensional rendering of a three-dimensional environment in this article. In order to achieve two-dimensional rendering of a three-dimensional environment, a transformation can be used to project the model data defining the three-dimensional environment. The transformation can generate a rendering so that the width, height, and depth of the three-dimensional object can be expressed on a flat screen using a vector projection from the model of the three-dimensional object to the viewpoint (e.g., a virtual camera position).

[0055] Figure 7 700 is a diagram illustrating aspects of a routine for controlling viewing and audio perspectives to focus on relevant activities of a communication session. It should be understood by those of ordinary skill in the art that the operations of the methods disclosed herein need not be presented in any particular order, and that performing some or all of the operations in an alternative order is possible and contemplated. For ease of description and illustration, the operations have been presented in the order in which they are presented. Operations may be added, omitted, performed together, and / or performed simultaneously without departing from the scope of the appended claims.

[0056] It should also be understood that the method shown can be started or ended at any time without being executed in full. Some or all of the operations of the method and / or substantially equivalent operations can be performed by executing computer-readable instructions contained on a computer storage medium defined herein. The term "computer-readable instructions" and its variations used in the specification and claims are widely used herein to include routines, applications, application modules, program modules, programs, components, data structures, algorithms, etc. Computer-readable instructions can be implemented on various system configurations, including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based programmable consumer electronics, combinations thereof, and the like. Although the exemplary routines described below are run on a system (e.g., one or more computing devices), it is understood that the routines can be executed on any computing system, which can include any number of computers working together to perform the operations disclosed herein.

[0057] It should be understood, therefore, that the logical operations described herein are implemented as a series of computer-implemented actions or program modules running on a computing system (such as the computing system described herein) and / or interconnected machine logic circuits or circuit modules within the computing system. The implementation is a matter of choice depending on the performance and other requirements of the computing system. Therefore, the logical operations may be implemented in software, firmware, dedicated digital logic, and any combination thereof.

[0058] in addition, Figure 7 The operations shown in the figures and other figures can be implemented in association with the exemplary user interfaces and systems described herein. For example, the various devices and / or modules described herein can generate, send, receive and / or display data associated with the content of a communication session (e.g., real-time content, broadcast events, recorded content, etc.) and / or a rendering UI including one or more participants of a remote computing device, avatars, channels, chat sessions, video streams, images, virtual objects and / or applications associated with the communication session.

[0059] Routine 700 includes operation 701, in which the system causes display of a first user interface arrangement 101A on a display device 629 associated with a user 10J. The first user interface arrangement 101A includes individual renderings of three-dimensional representations 251A-251E of multiple users 10A-10E participating in a communication session with user 10J. The first user interface arrangement 101A may also include 2D renderings of image files or real-time streams of the multiple users or other users. The individual three-dimensional representations 251A-251E have independent positions and independent orientations within the three-dimensional environment 200, and the independent positions and the independent orientations are each controlled by input data 650 provided by an associated user of the multiple users 10A-10E. Figure 1AAn example of a first user interface arrangement 101A is shown in FIG. In this example, the first user interface arrangement 101A has a rendering of a 3D avatar in a 3D virtual environment, and the viewing user is user J, a tenth user 10J.

[0060] At operation 703, the system can monitor the user activity of the communication session to identify at least one remote user 10A-10E that provides input or identifies user 10J. For example, the remote users 10A-10E of a threshold number may state the name of user J. In another example, the remote users 10A-10E of a threshold number can control their avatars to view the avatar of user J. In yet another example, the remote users 10A-10E of a threshold number can share data with user J, such as files or conference content. In some embodiments, one or more computers can monitor the input data 650 of multiple users 10A-10E from participating in the communication session 604 with user 10J. The system is configured to take one or more actions when the system detects the following items: identify or quote the input data 650 of user J in the gesture of the avatar, identify the input data 650 of user J in one or more operations for sharing content, and / or identify the input data 650 of user J in one or more forms of communication, wherein user J is identified as any name in verbal dialogue, text message, content, etc. For example, when a remote user mentions User J's name in a Word document using mention, the system may trigger one or more actions for controlling the visual or audio perspective.

[0061] At operation 705, the system may receive at least one user input from a remote user identifying user J. This may include input directing a gesture toward the user's avatar, for example, one of the avatars looking in the direction of the user, the remote user mentioning the user's name, the remote user sharing a file. Figure 1A In the example of , the system may receive input data 650 identifying user 10J.

[0062] In operation 705, the system may also receive a query from a user (user J). The query may include the name of any other individual in the virtual environment. By submitting queries about other users, the system may automatically highlight or focus on those users. For example, if user J provides voice input asking for Rita and Miguel, the system focuses on those users, such as Figure 1BAs shown. Additionally, input may also be provided by a computer. The computer may scan the profiles of each user in the 3D environment, and if two or more profiles have a matching score that satisfies a threshold, the system may focus on the 3D renderings associated with those matching profiles. This may include people on the same team, family members, managers, users with a certain management level associated with employees, etc. In another example, if user J has a set of hobbies, and other users (e.g., Rita and Miguel) have matching hobbies, the system may select from Figure 1A The user interface shown switches to Figure 1B The UI shown.

[0063] At operation 707, the system determines a subset of users from the plurality of users. In one example, if the first user 10A controls their avatar to view user J or if the first user 10A mentions user J's name in any document or communication, the system identifies the first user 10A as part of the subset of users. In another example, if input data 650 identifying user 10J is received from a plurality of computing devices 11A-11B associated with a plurality of users (e.g., user A and user B), the system determines that these users are part of the subset of users. In another illustrative example, if the first user 10A controls their avatar to view user J, the first user is identified as part of the subset of users.

[0064] The user subset may also be associated with the user providing the input data. For example, if the first user 10A controls their avatar to view user J, and the first user 10A has an avatar within a threshold distance of other avatars of other users (e.g., the second user 10B), then other users (e.g., the second user 10B) may also be identified as part of the user subset. In another example, if the first user 10A controls their avatar to view user J or shares content with user J, and the first user 10A has an avatar within a threshold distance of other avatars of other users or the other users are in a conversation with the first user 10A, then the system may identify other users as part of the user subset.

[0065] At operation 709, the system may transition the first user interface arrangement 101A to a second user interface arrangement 101B in response to input data 650 identifying the user 10J. The transition may include a first display of the first user interface arrangement 101A, the first user interface arrangement 101A including individual renderings of the three-dimensional representations 251A-251E of the plurality of users 10A-10E participating in the communication session 604, to the second user interface arrangement 101B focusing on a subset of one or more of the three-dimensional representations 251A-251B associated with the subset of users 10A-10B. The focusing on the subset of one or more of the three-dimensional representations 251A-251B generates a visual or audio control difference between the subset of one or more of the three-dimensional representations 251A-251B and other three-dimensional representations 251C-251E of other users 10C-10E of the plurality of users. For example, the system may increase the volume associated with the microphone of each user 10A-10B and turn down the microphones of the other users 10C-10E. Simultaneously or alternatively, the system may also increase the focus level on the three-dimensional representations 251A-251B associated with the subset of users 10A-10B and reduce or prevent the display of the representations 251C-251E of the other users 10C-10E in the plurality of users. The shifted focus may also include zooming in on the three-dimensional representations 251A-251B associated with the subset of users 10A-10B. Examples of such features are shown in FIG. Figure 1B , Figure 1C , Figure 2B , Figure 2C , Figure 3B and Figure 4B Shown in.

[0066] In some configurations, the operations described above may include a method for controlling a viewing perspective of an environment (200) to selectively focus on an activity associated with a user (10J) participating in a communication session (604), the method being configured to be executed on the system. For example, the system may display a 3D avatar or a 2D image of a user in a 3D virtual environment. The method may include additional operations for causing a first user interface arrangement (101A) to be displayed on a display device (629) associated with the user (10J), wherein the first user interface arrangement (101A) includes individual renderings of representations (151F-151J, 251A-251E) of a plurality of users (10A-10E) participating in a communication session with the user (10J), wherein each of the representations (151F-151J, 251A-251E) has an independent position and an independent orientation within the three-dimensional environment (200), the independent position and the independent orientation each being controlled by input data provided by an associated user of the plurality of users (10A-10E). The UI may include a 3D avatar and / or a 2D image of the user. The system may then monitor user activity of the communication session to identify at least one input from a remote user controlling one of the displayed avatars, looking for input identifying the user. Examples: (1) the remote user states the user's name, (2) the remote user controls their avatar to view the user, (3) the remote user shares data with the user. Operations may include monitoring input data (650) from multiple users (10A-10E) participating in a communication session (604) with the user (10J). The monitoring may identify any activity associated with the user. For example, operations may look for any preset conditions, which may include an audio signal identifying the user, any preset condition of users discussing certain topics specified in the preset conditions, for example, users discussing jazz or baseball, personnel profiles satisfying specific conditions of interest to the user for the meeting / interaction (members from a specific team, from a company, role, industry, etc.), eye gaze direction towards an area associated with the user, etc. When the system detects any input data (e.g., audio data or video data or any device or sensor data) indicating the presence of any preset condition (e.g., activity related to a user), the system can initiate any operation for focusing on a person or group of people. The data defining the preset condition can include topics of interest to a particular user, such as user J, a profile of a person, etc. When any topic identifying a person of interest to a user is input, the system can focus on the image of any user generating the input.

[0067] like Figure 5BAs shown, the system may receive at least one user input from a remote user, the user input directing a gesture toward an avatar of the user or mentioning the user's name, for example, one of the avatars looking in the direction of the user, the remote user mentioning the user's name, the remote user sharing a file. The operation may include determining that the input data (650) includes content that satisfies a preset condition associated with the user (10J), wherein the content that satisfies the preset condition associated with the user (10J) is received from one or more computing devices (11A-11B) associated with the subset of users (10A-10B).

[0068] like Figure 1B As shown, the new UI focuses on the subset of avatars that provide input. This can include focusing the UI on the avatars, for example, magnifying the relevant avatars and / or blurring other avatars. This can include the system converting a first user interface arrangement (101A) to a second user interface arrangement (101B) on a display device (629) associated with a user (10J) in response to determining that content of input data (650) satisfies a preset condition associated with the user (10J), the first user interface arrangement (101A) comprising individual renderings of representations (251A-251E) of a plurality of users (10A-10E) participating in a communication session (604), wherein the second user interface arrangement (101B) focuses on a subset of one or more of the representations (251A-251B) associated with the subset of users (10A-10B) by generating visual or audio control differences between the subset of one or more of the representations (251A-251B) and other representations (251C-251E) of other users (10C-10E) of the plurality of users.

[0069] Figure 8 600 is a diagram illustrating an exemplary environment 600 in which a system 602 can implement the techniques disclosed herein. It should be understood that the above-described subject matter can be implemented as a computer-controlled device, a computer process, a computing system, or an article of manufacture such as a computer-readable storage medium. The operations of the exemplary methods are represented by individual blocks and summarized with reference to these blocks. The methods are shown as a logical flow of blocks, each of which can represent one or more operations, which can be implemented in hardware, software, or a combination thereof. In the context of software, an operation represents a computer-executable instruction stored on one or more computer-readable media, which, when executed by one or more processors, enables the one or more processors to perform the operation.

[0070] Typically, computer executable instructions include routines, programs, objects, modules, components, data structures, etc. that perform specific functions or implement specific abstract data types. The order in which the operations are described should not be considered limiting, and any number of the operations may be performed in any order, combined in any order, subdivided into multiple sub-operations, and / or performed in parallel to implement the process. The process may be performed by resources associated with one or more devices, such as one or more internal or external CPUs or GPUs, and / or one or more hardware logic units, such as a field programmable gate array ("FPGA"), a digital signal processor ("DSP"), or other type of accelerator.

[0071] All of the above methods and processes may be embodied in, and fully automated via, software code modules executed by one or more general-purpose computers or processors. The code modules may be stored in any type of computer-readable storage medium or other computer storage device, such as the computer-readable storage medium or computer storage device described below. Some or all of the methods may alternatively be embodied in special-purpose computer hardware, such as the special-purpose computer hardware described below.

[0072] Any routine description, element or block in the flowcharts described herein and / or depicted in the accompanying drawings should be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing a specific logical function or element in the routine. Alternative implementations are included within the scope of the examples described herein, in which elements or functions may be deleted, or executed in a different order than that shown or discussed, including substantially simultaneously or in reverse order, depending on the functions involved as understood by those skilled in the art.

[0073] In some implementations, system 602 can be used to collect, analyze, and share data displayed to users of communication session 604. As shown, communication session 603 can be implemented between multiple client computing devices 606(1)-606(N) (where N is a number having a value of 2 or greater) associated with or as part of system 602. Client computing devices 606(1)-606(N) enable users (also referred to as individuals) to participate in communication session 603.

[0074] In this example, communication session 603 is hosted by system 602 over one or more networks 608. That is, system 602 can provide services that enable users of client computing devices 606(1) to 606(N) to participate in communication session 603 (e.g., via live viewing and / or recorded viewing). Thus, "participants" of communication session 603 can include users and / or client computing devices (e.g., multiple users can participate in a communication session in a room using a single client computing device), each of which can communicate with other participants. Alternatively, communication session 603 can be hosted by one of client computing devices 606(1) to 606(N) utilizing peer-to-peer technology. System 602 can also host chat conversations and other team collaboration functionality (e.g., as part of an application suite).

[0075] In some embodiments, such chat conversations and other team collaboration functions are considered external communication sessions that are distinct from the communication session 603. The computing system 602 that collects participant data in the communication session 603 may be able to link to such external communication sessions. Thus, the system may receive information, such as date, time, session details, etc., that enables connection to such external communication sessions. In one example, a chat conversation may be conducted in accordance with the communication session 603. Additionally, the system 602 may host a communication session 603 that includes at least a plurality of participants that are co-located at a meeting location (e.g., a conference room or auditorium) or located at different locations.

[0076] In the examples described herein, client computing devices 606(1) to 606(N) participating in a communication session 603 are configured to receive and render communication data for display on a user interface of a display screen. The communication data may include a collection of various instances or streams of real-time content and / or recorded content. The collection of various instances or streams of real-time content and / or recorded content may be provided by one or more cameras (e.g., video cameras). For example, individual streams of real-time content or recorded content may include media data associated with a video feed provided by a camera (e.g., audio and visual data capturing the appearance and voice of users participating in the communication session). In some embodiments, the video feed may include such audio and visual data, one or more still images, and / or one or more avatars. The one or more still images may also include one or more avatars.

[0077] Another example of an individual stream of real-time content or recorded content may include media data including avatars of users participating in a communication session and audio data capturing the users' voices. Yet another example of an individual stream of real-time content or recorded content may include media data including a file displayed on a display screen and audio data capturing the users' voices. Thus, individual streams of real-time content or recorded content within the communication data enable remote conferencing between a group of people and content sharing within the group of people. In some embodiments, individual streams of real-time content or recorded content within the communication data may originate from multiple co-located cameras located in a space such as a room for recording or live streaming a presentation including one or more individuals making a presentation and one or more individuals consuming the presented content.

[0078] Participants or attendees can view the content of communication session 603 in real time as the activity occurs, or alternatively, view the content of communication session 603 via recording at a later time after the activity occurs. In the examples described herein, client computing devices 606(1) to 606(N) participating in communication session 603 are configured to receive and render communication data for display on a user interface of a display screen. The communication data may include a collection of various instances or streams of real-time content and / or recorded content. For example, an individual stream of content may include media data associated with a video feed (e.g., audio and visual data that captures the appearance and voice of users participating in the communication session). Another example of an individual stream of content may include media data that includes an avatar of a user participating in the conference session and audio data that captures the user's voice. Yet another example of an individual stream of content may include media data that includes content items displayed on a display screen and / or audio data that captures the user's voice. Thus, the individual streams of content within the communication data enable a group of people dispersed in remote locations to conveniently hold a meeting or broadcast a presentation.

[0079] A participant or attendee of a communication session is a person who is within range of a camera or other image and / or audio capture device, so that the person's movements and / or sounds generated while the person is viewing and / or listening to content shared via the communication session can be captured (e.g., recorded). For example, a participant may be sitting in a crowd, viewing shared content in real time at a broadcast location where a stage presentation is taking place. Alternatively, a participant may be sitting in an office conference room, viewing the shared content of a communication session with other colleagues via a display screen. Even more, a participant may be alone in his or her office or home, sitting or standing in front of a personal device (e.g., a tablet device, a smartphone, a computer, etc.) viewing the shared content of a communication session.

[0080] Figure 8System 602 includes device 610. Device 610 and / or other components of system 602 may include distributed computing resources that communicate with each other and / or with client computing devices 606(1) to 606(N) via one or more networks 608. In some examples, system 602 may be a stand-alone system that is tasked with managing aspects of one or more communication sessions (e.g., communication session 603). As an example, system 602 may be managed by an entity such as Slack, WEBEX, GOTOMEETING, GOOGLEHANGOUTS, etc.

[0081] The network 608 may include, for example, a public network (e.g., the Internet), a private network (e.g., an institutional and / or personal intranet), or some combination of a private network and a public network. The network 608 may also include any type of wired and / or wireless network, including but not limited to a local area network ("LAN"), a wide area network ("WAN"), a satellite network, a wired network, a Wi-Fi network, a WiMax network, a mobile communication network (e.g., 3G, 4G, etc.), or any combination thereof. The network 608 may use communication protocols, including packet-based and / or datagram-based protocols, such as the Internet Protocol ("IP"), the Transmission Control Protocol ("TCP"), the User Datagram Protocol ("UDP"), or other types of protocols. In addition, the network 608 may also include multiple devices that facilitate network communications and / or form the basis of network hardware, such as switches, routers, gateways, access points, firewalls, base stations, repeaters, backbone devices, etc.

[0082] In some examples, network 608 may also include devices that enable connection to a wireless network, such as a wireless access point ("WAP"). Examples support connection via a WAP that sends and receives data via various electromagnetic frequencies (e.g., wireless frequencies), including WAPs that support the Institute of Electrical and Electronics Engineers ("IEEE") 802.11 standards (e.g., 802.11g, 802.11n, 802.11ac, etc.) and other standards.

[0083] In various examples, device 610 may include one or more computing devices that operate in a cluster or other grouping configuration to share resources, balance loads, improve performance, provide failover support or redundancy, or for other purposes. For example, device 610 may belong to various types of devices, such as traditional server type devices, desktop computer type devices, and / or mobile type devices. Therefore, although shown as a single type of device or server type device, device 610 may include a variety of device types and is not limited to a particular type of device. Device 610 may represent, but is not limited to, a server computer, a desktop computer, a web server computer, a personal computer, a mobile computer, a laptop computer, a tablet computer, or any other type of computing device.

[0084] The client computing device (e.g., one of client computing devices 606(1)-606(N)) (each client computing device also referred to herein as a "data processing system") can be of various types of devices, which can be the same as or different from device 610, such as a traditional client type device, a desktop type device, a mobile type device, a dedicated type device, an embedded type device, and / or a wearable type device. Thus, the client computing device can include, but is not limited to, a desktop computer, a gaming console and / or gaming device, a tablet computer, a personal data assistant ("PDA"), a mobile phone / tablet hybrid device, a laptop computer, a telecommunications device, a computer navigation type client computing device (e.g., a satellite-based navigation system, including a global positioning system ("GPS") device), a wearable device, a virtual reality ("VR") device, an augmented reality ("AR") device, an implanted computing device, an automotive computer, a network television, a thin client, a terminal, an Internet of Things ("IoT") device, a workstation, a media player, a personal video recorder ("PVR"), a set-top box, a camera, an integrated component (e.g., a peripheral device) for inclusion in a computing device, an appliance, or any other type of computing device. Furthermore, the client computing device may include a combination of the previously listed client computing device examples, such as, for example, a desktop computer-type device or a mobile-type device combined with a wearable device, etc.

[0085] Client computing devices 606(1) to 606(N) of various classes and device types may represent any type of computing device having one or more data processing units 692 operably connected to a computer-readable medium 694, for example, via a bus 616, which in some cases may include a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and one or more of any variety of local, peripheral, and / or independent buses.

[0086] Executable instructions stored on computer-readable medium 694 may include, for example, operating system 619 , client module 620 , profile module 622 , and other modules, programs, or applications that may be loaded and executed by data processing unit 692 .

[0087] The client computing devices 606(1)-606(N) may also include one or more interfaces 624 to enable communication between the client computing devices 606(1)-606(N) and other networked devices (e.g., device 610) over the network 608. Such network interfaces 624 may include one or more network interface controllers (NICs) or other types of transceiver devices for sending and receiving communications and / or data over the network. In addition, the client computing devices 606(1)-606(N) may include input / output ("I / O") interfaces (devices) 626 that enable communication with input / output devices, such as user input devices, including peripheral input devices (e.g., game controllers, keyboards, mice, pens, voice input devices (e.g., microphones), cameras for acquiring and providing video feeds and / or still images, touch input devices, gesture input devices, etc.) and / or output devices, including peripheral output devices (e.g., displays, printers, audio speakers, tactile output devices, etc.). Figure 8 Client computing device 606 ( 1 ) is shown connected in some manner to a display device (eg, display screen 629 (N)) that can display a UI according to the techniques described herein.

[0088] exist Figure 8 In the exemplary environment 600 of FIG. 6 , client computing devices 606 ( 1 ) to 606 (N) can use their respective client modules 620 to connect to each other and / or other external devices to participate in a communication session 603 or to contribute activities to a collaborative environment. For example, a first user can use client computing device 606 ( 1 ) to communicate with a second user of another client computing device 606 ( 2 ). When executing client module 620, the users can share data, which can cause client computing device 606 ( 1 ) to connect to system 602 and / or other client computing devices 606 ( 2 ) to 606 (N) via network 608 .

[0089] The client computing devices 606(1) to 606(N) may use their respective profile modules 622 to generate participant profiles ( Figure 8602) and provide the participant profile to other client computing devices and / or devices 610 of system 602. The participant profile may include one or more of an identity of a user or group of users (e.g., name, unique identifier (“ID”), etc.), user data such as personal data (e.g., machine data such as location (e.g., IP address, room in a building, etc.), and technical capabilities, etc. The participant profile may be used to register participants for a communication session.

[0090] like Figure 8 As shown, device 610 of system 602 includes server module 630 and output module 632. In this example, server module 630 is configured to receive media streams 634(1) to 634(N) from individual client computing devices (e.g., client computing devices 606(1) to 606(N)). As described above, the media streams may include video feeds (e.g., audio and visual data associated with a user), audio data to be output along with a presentation of an avatar of the user (e.g., an audio-only experience that does not transmit the user's video data), text data (e.g., a text message), file data, and / or screen sharing data (e.g., a document, a slideshow, an image, a video displayed on a display screen, etc.), etc. Thus, server module 630 is configured to receive a collection of various media streams 634(1) to 634(N) (the collection referred to herein as "media data 634") during a real-time viewing communication session 603. In some scenarios, not all client computing devices participating in communication session 603 provide media streams. For example, the client computing device may be a consuming device or a “listening” device only, such that it only receives content associated with the communication session 603 but does not provide any content to the communication session 603 .

[0091] In various examples, the server module 630 can select aspects of the media stream 634 to be shared with individual client computing devices of the participating client computing devices 606(1) to 606(N). Thus, the server module 630 can be configured to generate session data 636 based on the stream 634 and / or pass the session data 636 to the output module 632. The output module 632 can then transmit communication data 639 to the client computing devices (e.g., the client computing devices 606(1) to 606(3) participating in the real-time viewing communication session). The communication data 639 can include video, audio, and / or other content data provided by the output module 632 based on content 650 associated with the output module 632 and based on the received session data 636. The content 650 can include the stream 634 or other shared data, such as an image file, a spreadsheet file, a slide show, a document, etc. The stream 634 can include a video component that depicts an image captured by the I / O device 626 on each client computer. The content 650 also includes input data from each user, which can be used to control the direction and position of the presentation. The content may also include instructions for sharing the data and an identifier of a recipient of the shared data. Thus, content 650 is also referred to herein as input data 650 or input 650.

[0092] As shown, output module 632 sends communication data 639(1) to client computing device 606(1), sends communication data 639(2) to client computing device 606(2), sends communication data 639(3) to client computing device 606(3), etc. The communication data 639 sent to the client computing devices may be the same or may be different (e.g., the positioning of the flow of content within the user interface may vary from device to device).

[0093] In various implementations, the device 610 and / or the client module 620 may include a GUI presentation module 640. The GUI presentation module 640 may be configured to analyze the communication data 639 to be transmitted to one or more client computing devices 606. Specifically, the UI presentation module 640 at the device 610 and / or the client computing device 606 may analyze the communication data 639 to determine an appropriate manner to display the video, image, and / or content on the display screen 629 of the associated client computing device 606. In some implementations, the GUI presentation module 640 may provide the video, image, and / or content to a presentation GUI 646 rendered on the display screen 629 of the associated client computing device 606. The GUI presentation module 640 may cause the presentation GUI 646 to be rendered on the display screen 629. The presentation GUI 646 may include the video, image, and / or content analyzed by the GUI presentation module 640.

[0094] In some implementations, the presentation GUI 646 may include multiple sections or grids that may render or include video, images, and / or content for display on the display screen 629. For example, a first section of the presentation GUI 646 may include a video feed of a presenter or individual, and a second section of the presentation GUI 646 may include a video feed of an individual consuming conference information provided by the presenter or individual. The GUI presentation module 640 may populate the first section and the second section of the presentation GUI 646 in a manner that appropriately mimics an environmental experience that the presenter and individual may share.

[0095] In some embodiments, the GUI presentation module 640 can zoom in or provide a zoomed view of the individual represented by the video feed to highlight the individual's reaction to the presenter, such as facial features. In some embodiments, the presentation GUI 646 can include video feeds of multiple participants associated with a meeting (e.g., a general communication session). In other embodiments, the presentation GUI 646 can be associated with a channel, such as a chat channel, an enterprise team channel, etc. Thus, the presentation GUI 646 can be associated with an external communication session that is different from a general communication session.

[0096] Fig. 9 A diagram showing exemplary components of an exemplary device 700 (also referred to herein as a "computing device") that is configured to generate data for some user interfaces disclosed herein. The device 700 can generate data that can include one or more sections that can render or include video, images, virtual objects, and / or content for display on a display screen 629. The device 700 can represent one of the devices described herein. Additionally or alternatively, the device 700 can represent one of the client computing devices 606.

[0097] As shown, device 700 includes one or more data processing units 702, computer-readable media 704, and communication interface 706. The components of device 700 may be operatively connected, for example, via bus 709, which may include one or more of a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and any kind of local, peripheral, and / or independent bus.

[0098] As utilized herein, a data processing unit (e.g., data processing unit 702 and / or data processing unit 692) may represent, for example, a CPU-type data processing unit, a GPU-type data processing unit, a field programmable gate array ("FPGA"), another type of DSP, or other hardware logic components that may be driven by a CPU in some cases. For example, but not limited to, illustrative types of hardware logic components that may be utilized include application specific integrated circuits ("ASICs"), application specific standard products ("ASSPs"), systems on chips ("SOCs"), complex programmable logic devices ("CPLDs"), and the like.

[0099] As used herein, computer-readable media (e.g., computer-readable media 704 and computer-readable media 694) can store instructions that can be executed by a data processing unit. The computer-readable medium can also store instructions that can be executed by an external data processing unit (e.g., an external CPU, an external GPU) and / or instructions that can be executed by an external accelerator (e.g., an FPGA-type accelerator, a DSP-type accelerator, or any other internal or external accelerator). In various examples, at least one CPU, GPU, and / or accelerator is incorporated into the computing device, while in some examples, one or more of the CPU, GPU, and / or accelerator is located outside the computing device.

[0100] Computer readable media (also referred to herein as computer readable media) may include computer storage media and / or communication media. Computer storage media may include one or more of volatile memory, non-volatile memory and / or other persistent and / or secondary computer storage media, removable and non-removable computer storage media, implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Thus, computer storage media includes media in tangible and / or physical form contained in a device and / or in a hardware component that is part of the device or external to the device, including, but not limited to, random access memory ("RAM"), static random access memory ("SRAM"), dynamic random access memory ("DRAM"), phase change memory ("PCM"), read-only memory ("ROM"), erasable programmable read-only memory ("EPROM"), electrically erasable programmable read-only memory ("EEPROM"), flash memory, compact disc read-only memory ("CD-ROM"), digital versatile disc ("DVD"), optical cards or other optical storage media, magnetic cassettes, magnetic tape, magnetic disk storage devices, magnetic cards or other magnetic storage devices or media, solid-state memory devices, storage arrays, network attached storage devices, storage area networks, hosted computer storage devices, or any other storage memory, storage device, and / or storage medium that can be used to store and maintain information for access by a computing device. Computer storage media may also be referred to herein as computer-readable storage media, non-transitory computer-readable storage media, non-transitory computer-readable media, or computer storage media.

[0101] In contrast to computer storage media, communication media may embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal (e.g., carrier wave) or other transport mechanism. As defined herein, computer storage media does not include communication media. That is, computer storage media does not include communication media that consists solely of a modulated data signal, carrier wave, or propagation signal itself.

[0102] The communication interface 706 may represent, for example, a network interface controller ("NIC") or other type of transceiver device for sending and receiving communications over a network. In addition, the communication interface 706 may include one or more cameras and / or audio devices 722 to generate video feeds and / or still images, etc.

[0103] In the illustrated example, computer-readable medium 704 includes a data repository 708. In some examples, data repository 708 includes a data storage device, such as a database, a data warehouse, or other type of structured or unstructured data storage device. In some examples, data repository 708 includes a corpus and / or a relational database having one or more tables, indexes, stored procedures, etc. to enable data access, including, for example, one or more of a Hypertext Markup Language ("HTML") table, a Resource Description Framework ("RDF") table, a Web Ontology Language ("OWL") table, and / or an Extensible Markup Language ("XML") table.

[0104] The data repository 708 may store data for the operation of processes, applications, components, and / or modules stored in the computer-readable medium 704 and / or executed by the data processing unit 702 and / or accelerator. For example, in some examples, the data repository 708 may store session data 710 (e.g., Figure 8 The data repository 708 may also include context data 714, such as content including video, audio, or other content to be rendered and displayed on one or more display screens 629. The hardware data 711 may define aspects of any device, such as multiple display screens of a computer. The context data 714 may define any type of activity or state associated with individual users 10A-10L, each of which is associated with an individual video stream in the multiple video streams 634. For example, the context data may define a person's level in an organization, the relationship between each person's level and the levels of other people, a person's performance level, or any other activity or state information that may be used to determine the location of a person's rendering within the virtual environment. This contextual information can also be fed into any model to help emphasize key words spoken by a person of a certain level, highlight the UI when background voices of a person of a certain level are detected, or change the emotion display in a specific way when a person of a certain level is detected to have a certain emotion.

[0105] Alternatively, some or all of the above data may be stored in a separate memory 716 on one or more data processing units 702, such as a CPU-type processor, a GPU-type processor, an accelerator of an FPGA type, an accelerator of a DSP type, and / or a memory on another accelerator. In this example, the computer-readable medium 704 also includes an operating system 718 and an application programming interface 710 (API) configured to expose the functions and data of the device 700 to other devices. In addition, the computer-readable medium 704 includes one or more modules, such as a server module 730, an output module 732, and a GUI rendering module 740, but the number of modules shown is only an example, and the number may vary. That is, the functions described in association with the modules shown herein may be performed by a smaller number of modules or a larger number of modules on a single device, or distributed on multiple devices.

[0106] The following terms supplement this disclosure.

[0107] Item A: A method for controlling a viewing perspective of a three-dimensional environment (200) to follow activities associated with a user (10J) participating in a communication session (604), the method being configured to be executed on a system, the method comprising: causing display of a first user interface arrangement (101A) on a display device (629) associated with the user (10J), wherein the first user interface arrangement (101A) comprises individual renderings of three-dimensional representations (251A-251E) of a plurality of users (10A-10E) participating in the communication session with the user (10J), wherein each of the three-dimensional representations (251A-251E) has an independent position and an independent orientation within the three-dimensional environment (200), the independent position and the independent orientation each being controlled by input data provided by an associated user of the plurality of users (10A-10E); Figure 1AAn example is shown in: Displaying a 3D avatar in a 3D virtual environment: the user is a user of a computer viewing the display; monitoring input data (650) from a plurality of users (10A-10E) participating in a communication session (604) with the user (10J), wherein the system monitors the input data (650) to determine when the input data (650) identifies the user (10J); monitoring user activity of the communication session to identify at least one input from a remote user controlling one of the displayed avatars, looking for input identifying the user. Examples: (1) the remote user states the user's name, (2) the remote user controls their avatar to view the user, (3) the remote user shares data with the user; receiving input data (650) identifying the user (10J), wherein the input data (650) identifies the user (10J) received from one or more computing devices (11A-11B) associated with a subset of users (10A-10B), each computing device being associated with a subset of one or more of the three-dimensional representations (251A-251B); Figure 5B An example is shown in which the system can receive at least one user input from a remote user, the user input directing a gesture toward an avatar of the user or mentioning the user's name, such as one of the avatars looking in the direction of the user, the remote user mentioning the user's name, the remote user sharing a file; and in response to input data (650) identifying the user (10J), transforming a first user interface arrangement (101A) including individual renderings of three-dimensional representations (251A-251E) of multiple users (10A-10E) participating in a communication session (604) to a second user interface arrangement The invention relates to a method of controlling a plurality of three-dimensional representations of a user interface (101B), wherein the second user interface arrangement (101B) focuses on a subset of one or more of the three-dimensional representations (251A-251B) associated with the subset of users (10A-10B), wherein the focusing on the subset of one or more of the three-dimensional representations (251A-251B) generates a visual or audio control difference between the subset of one or more of the three-dimensional representations (251A-251B) and other three-dimensional representations (251C-251E) of other users (10C-10E) of the plurality of users, examples of which are described in Figure 1B Shown in: The new UI focuses on the subset of avatars providing input. This can include an avatar-focused UI, which means zooming in on the relevant avatar and / or blurring other avatars.

[0108] Clause B: A method as defined in any other clause, wherein the second user interface arrangement focuses on a subset of the one or more representations in the three-dimensional representations by zooming in to include an enlarged rendering of the subset of the one or more representations in the three-dimensional representations and excluding viewing perspectives of other three-dimensional representations of other users of the plurality of users. Figure 2B Shown in: Focus view via zoom: The viewing perspective zooms in on the subject and excludes other users who are not in focus.

[0109] Item C: A method according to any other item, wherein the second user interface arrangement focuses on a subset of one or more representations in the three-dimensional representations by obscuring the rendering of other three-dimensional representations displayed in the first user interface arrangement, wherein the other three-dimensional representations exclude the subset of one or more representations in the three-dimensional representations. Figure 1B Vaguely other incarnations were on display.

[0110] Clause D: A method as described in any other clause, wherein the input data directs the position or orientation of one or more representations of the subset of users in the three-dimensional representation towards the representation of the user. Figures 5A-5C : Other users view this user.

[0111] Clause E: A method according to any other clause, wherein the input data includes at least one audio signal, the audio signal including a voice conversation identifying the user, and wherein transitioning the first user interface arrangement to the second user interface arrangement is invoked in response to receiving the audio signal including a voice conversation identifying the user. Figure 1A-1B , Figure 2A-2B : Mention the user in a conversation.

[0112] Clause E: A method according to any other clause, wherein the input for causing the transformation includes input data that directs the position or orientation of a threshold number of three-dimensional representations of multiple users toward the representation of the user, wherein the transformation is invoked in response to determining that the input directs the position or orientation of a threshold number of three-dimensional representations of multiple users toward the representation of the user, and wherein the subset of three-dimensional representations is selected when the subset of three-dimensional representations are within a predetermined distance of each other.

[0113] Clause F: A method according to any other clause, wherein the input for causing the transition includes an audio signal identifying the user, wherein the transition is invoked in response to determining that the input includes an audio signal identifying the user, and wherein the subset of three-dimensional representations is selected when the subsets of three-dimensional representations are within a predetermined distance of each other.

[0114] Finally, although various configurations have been described using language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended expressions is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claimed subject matter.

Claims

1. A method for controlling a viewing perspective of an environment to selectively focus on relevant activities of users participating in a communication session, the method being configured to be executed on a system, the method comprising: causing display of a first user interface arrangement on a display device associated with the user, wherein the first user interface arrangement includes individual renderings of representations of a plurality of users participating in the communication session with the user, wherein each of the representations has an independent position and an independent orientation within a three-dimensional environment, the independent position and the independent orientation each being controlled by input data provided by an associated user of the plurality of users; monitoring input data from said plurality of users participating in said communication session with said user; determining that the input data includes content that satisfies a preset condition associated with the user, wherein the content that satisfies the preset condition associated with the user is received from one or more computing devices associated with a subset of users; and In response to determining that the content of the input data satisfies the preset condition associated with the user, the first user interface arrangement including individual renderings of representations of the multiple users participating in the communication session is transformed to a second user interface arrangement on the display device associated with the user by generating a visual or audio control difference between a subset of one or more representations in the representations associated with the subset of users and other representations of other users among the multiple users, wherein the second user interface arrangement is focused on a subset of one or more representations in the representations associated with the subset of users.

2. The method according to claim 1, wherein: The second user interface arrangement focuses on a subset of the one or more representations in the representations by zooming in to include a magnified rendering of the subset of the one or more representations in the representations and excluding a viewing perspective of other representations of other users of the plurality of users, wherein the subsets of users are each associated with a subset of the one or more representations in the representations.

3. The method according to claim 1, wherein: The second user interface arrangement focuses on the subset of the one or more of the representations by obscuring rendering of other representations displayed in the first user interface arrangement, wherein the other representations exclude the subset of the one or more of the representations.

4. The method according to claim 1, wherein: The input data directs the position or orientation of the one or more representations of the subset of users in the representation towards the representation of the user.

5. The method according to claim 1, wherein: The input data comprises at least one audio signal comprising a voice dialogue identifying the user, wherein transitioning the first user interface arrangement to the second user interface arrangement is invoked in response to receiving the audio signal comprising a voice dialogue identifying the user.

6. The method according to claim 1, wherein: The input for causing the transition includes input data that directs the position or orientation of a threshold number of representations of the plurality of users toward a representation of the user, wherein the transition is invoked in response to determining that the input directs the position or orientation of the threshold number of representations of the plurality of users toward a representation of the user, wherein the subset of representations is selected when the subset of representations is within a predetermined distance of each other.

7. The method according to claim 1, wherein: The input for causing the transition includes an audio signal identifying the user, wherein the transition is invoked in response to determining that the input includes the audio signal identifying the user, wherein the subset of representations is selected when the subset of representations are within a predetermined distance of each other.

8. A system for controlling a viewing perspective of an environment to focus a user interface display on relevant activities of users participating in a communication session, the system comprising: one or more processing units; as well as A computer-readable storage medium having encoded thereon computer-executable instructions for causing the one or more processing units to perform a method comprising: causing display of a first user interface arrangement on a display device associated with the user, wherein the first user interface arrangement includes individual renderings of representations of a plurality of users participating in the communication session with the user, wherein each of the representations has an independent position and an independent orientation within a three-dimensional environment, the independent position and the independent orientation each being controlled by input data provided by an associated user of the plurality of users; monitoring input data from said plurality of users participating in said communication session with said user; determining that the input data includes content that satisfies a preset condition associated with the user, wherein the content that satisfies the preset condition associated with the user is received from one or more computing devices associated with a subset of users; and In response to determining that the content of the input data satisfies the preset condition associated with the user, the first user interface arrangement including individual renderings of representations of the multiple users participating in the communication session is transformed to a second user interface arrangement on the display device associated with the user by generating a visual or audio control difference between a subset of one or more representations in the three-dimensional representation and other representations of other users among the multiple users, wherein the second user interface arrangement is focused on a subset of one or more representations in the representation associated with the subset of users.

9. The system according to claim 8, wherein: The second user interface arrangement focuses on the subset of the one or more of the representations by zooming in to include a zoomed-in rendering of the subset of the one or more of the representations and excluding viewing perspectives of other representations of other users of the plurality of users.

10. The system according to claim 8, wherein: The second user interface arrangement focuses on the subset of the one or more of the representations by obscuring rendering of other representations displayed in the first user interface arrangement, wherein the other representations exclude the subset of the one or more of the representations.

11. The system according to claim 8, wherein: The input data directs the position or orientation of the one or more representations of the subset of users in the representation towards the representation of the user.

12. The system according to claim 8, wherein: The input data comprises at least one audio signal comprising a voice dialogue identifying the user, wherein transitioning the first user interface arrangement to the second user interface arrangement is invoked in response to receiving the audio signal comprising a voice dialogue identifying the user.

13. The system according to claim 8, wherein: The input for causing the transition includes input data that directs the position or orientation of a threshold number of representations of the plurality of users toward a representation of the user, wherein the transition is invoked in response to determining that the input directs the position or orientation of the threshold number of representations of the plurality of users toward a representation of the user, wherein the subset of representations is selected when the subset of representations is within a predetermined distance of each other.

14. The system according to claim 8, wherein: The input for causing the transition includes an audio signal identifying the user, wherein the transition is invoked in response to determining that the input includes the audio signal identifying the user, wherein the subset of representations is selected when the subset of representations are within a predetermined distance of each other.

Images